Method and system for implementing augmented reality (AR)-based assistance within work environment

ABSTRACT

Novel tools and techniques are provided for implementing augmented reality (AR)-based assistance within a work environment. In various embodiments, a computing system might receive, from a camera having a field of view of a work environment, first images of at least part of the work environment, the first images overlapping with a field of view of a user wearing an AR headset; might analyze the received first images to identify objects; might query a database(s) to determine a task associated with a first object(s) among the identified objects; might generate an image overlay providing at least one of graphical icon-based, text-based, image-based, and/or highlighting-based instruction(s) each indicative of instructions presented to the user to implement the task associated with the first object(s); and might display, to the user&#39;s eyes through the AR headset, the generated first image overlay that overlaps with the field of view of the user&#39;s eyes.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to U.S. Patent Application Ser. No.62/675,122 (the “'122 Application”), filed May 22, 2018 by Amir Ben-Doret al., entitled, “Method and Apparatus for Facilitating Manual Sortingof Objects,” the disclosure of which is incorporated herein by referencein its entirety for all purposes.

The respective disclosures of these applications/patents (which thisdocument refers to collectively as the “Related Applications”) areincorporated herein by reference in their entirety for all purposes.

COPYRIGHT STATEMENT

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD

The present disclosure relates, in general, to methods, systems, andapparatuses for implementing assistance within a work environment, and,more particularly, to methods, systems, and apparatuses for implementingaugmented reality (AR)-based assistance within a work environment.

BACKGROUND

In conventional work environments, such as in cases where tasks involveworking with a large number of objects—particularly where such objectsare difficult to distinguish from other similar objects at a glance—orwhere automation is difficult to implement due to fragility of suchobjects and/or due to many different tasks being required (thus makingautomation expensive or cumbersome to implement), user performance ofthe tasks (such as by hand or using hand tools) can result ininefficiencies or inaccuracies in the performance of the tasks.

For example, tasks involving hand sorting of objects, which may berequired in a number of areas of technology, might include hand sortingof tissue samples that are stained and mounted on microscope slides forviewing by pathologists in pathology laboratories. The tissue samplesmay be sorted into containers that are sent to the pathologists whorequested those stained samples. The number of slides processed in thismanner in a day at a large medical facility can be in the thousands,serving many pathologists, and hence, it would be advantageous toprovide an automated sorting system. Unfortunately, the slides areusually irreplaceable because the patient is no longer in the medicalfacility or the tissue from which the slides were made is no longeravailable. Accordingly, the slides have to be sorted by hand into thecontainers to avoid damage.

If the volume of objects that need to be sorted is small, providing anautomated solution presents challenges, because the cost of themachinery cannot be recouped. Similarly, if the objects are small,fragile, or of a number of different shapes, the cost of fully automatedsorting machinery becomes prohibitive. In addition, automated systemstend to require dedicated floor or desk space. In applications in whichthe system is used only occasionally, the cost of setting up the systemand then taking it down can be prohibitive. Finally, if the volume ofsorting varies significantly from day to day, on some days, it may benecessary to use multiple technicians for the sorting. However, if thesorting apparatus requires a fixed setup, increasing the capacity byadding personnel presents challenges.

Conventional systems that could be used to sort these systems or toassist a user in the hand sorting of the slides may require a physicalsuper structure to hold its components in place. Hence, the system isnot easily moved, nor is it well suited for simultaneous or concurrentuse by multiple technicians.

Hence, there is a need for more robust and scalable solutions forimplementing assistance within a work environment, and, moreparticularly, to methods, systems, and apparatuses for implementingaugmented reality (AR)-based assistance within a work environment.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of particularembodiments may be realized by reference to the remaining portions ofthe specification and the drawings, in which like reference numerals areused to refer to similar components. In some instances, a sub-label isassociated with a reference numeral to denote one of multiple similarcomponents. When reference is made to a reference numeral withoutspecification to an existing sub-label, it is intended to refer to allsuch multiple similar components.

FIG. 1 is a schematic diagram illustrating a system for implementingaugmented reality (AR)-based assistance within a work environment, inaccordance with various embodiments.

FIGS. 2A and 2B are schematic diagrams illustrating a set ofnon-limiting examples of sorting, categorizing, arranging, or organizingof objects performed by a user utilizing an AR headset that is used forimplementing AR-based assistance within a work environment, inaccordance with various embodiments.

FIGS. 3A and 3B are schematic diagrams illustrating a set ofnon-limiting examples of sorting, categorizing, arranging, or organizingof objects performed by a plurality of users by utilizing AR headsetsthat are used for implementing AR-based assistance within a workenvironment, in accordance with various embodiments.

FIGS. 4A and 4B are flow diagrams illustrating a method for implementingAR-based assistance within a work environment, in accordance withvarious embodiments.

FIG. 5 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments.

FIG. 6 is a block diagram illustrating a networked system of computers,computing systems, or system hardware architecture, which can be used inaccordance with various embodiments.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

Overview

Various embodiments provide tools and techniques for implementingassistance within a work environment, and, more particularly, tomethods, systems, and apparatuses for implementing augmented reality(AR)-based assistance within a work environment.

In various embodiments, a computing system might receive, from one ormore first image capture devices having a field of view of at least afirst portion of a work environment (i.e., directed in front of an ARheadset worn by a first user), one or more first images of the at leasta first portion of the work environment, the one or more first imagesoverlapping with a field of view of eyes of the first user wearing theAR headset. The computing system might analyze the received one or morefirst images to identify one or more first objects disposed in the workenvironment; might query at least one database to determine a first taskassociated with at least one first object among the identified one ormore first objects; might generate a first image overlay, the firstimage overlay providing one or more instructions (which might include,without limitation, at least one of a graphical icon-based instruction,a text-based instruction, an image-based instruction, or ahighlighting-based instruction, and/or the like) to be presented to auser to implement the first task associated with the at least one firstobject; and might display, to the eyes of the first user through the ARheadset, the generated first image overlay that overlaps with the fieldof view of the eyes of the first user.

According to some embodiments, the computing system might include,without limitation, one of a processor disposed in the AR headset, acomputing system disposed in the work environment, a remote computingsystem disposed external to the work environment and accessible over anetwork, or a cloud computing system, and/or the like. In some cases,the AR headset might include, but is not limited to, one of a set of ARgoggles, a pair of AR-enabled eyewear, an AR-enabled smartphone mountedin a headset, or an AR helmet, and/or the like. In some instances, atleast one of the one or more first image capture devices might bedisposed on a portion of a housing of the AR headset.

In some instances, the at least one of the graphical icon-basedinstruction, the text-based instruction, the image-based instruction, orthe highlighting-based instruction might include, but is not limited to,at least one of a graphical icon-based instruction, a text-basedinstruction, an image-based instruction, or a highlighting-basedinstruction that, when displayed within the generated first imageoverlay, is superimposed over, displayed around, or displayed beside theat least one first object as viewed by the user through the AR headset,and/or the like. In some cases, the graphical icon-based instructionmight include, without limitation, at least one of a graphical iconrepresenting identification information associated with the at least onefirst object; a graphical icon representing identification informationassociated with a second object with which the at least one first objectis intended to interact as part of the first task; a colored graphicalicon distinguishing one first object from another first object among theat least one first object; a graphical icon comprising a directionalarrow representing a direction that the at least one first object shouldbe taken as part of the first task; or a colored graphical iconcomprising a colored directional arrow distinguishing a first directionthat the at least one first object should be taken as part of the firsttask from a second direction that a third object should be taken as partof the first task, and/or the like. In some instances, the text-basedinstruction might include, but is not limited to, at least one of afloating text window comprising textual instructions corresponding tothe first task; or a surface text window that is superimposed on one ofa table-top surface, a wall surface, or an object surface and thatcomprises textual instructions corresponding to the first task. In somecases, the floating text window, when displayed within the generatedfirst image overlay, is displayed as a floating image beside the atleast one first object or displayed as a floating image within the fieldof view of the eyes of the first user.

In some instances, the image-based instruction might include, withoutlimitation, at least one of an image representing identificationinformation associated with the at least one first object; an imagerepresenting identification information associated with a second objectwith which the at least one first object is intended to interact as partof the first task; an image comprising a directional arrow representinga direction that the at least one first object should be taken as partof the first task; an image comprising images of numbers or codesrepresenting an order of processes of the first task associated with theat least one first object; a magnified image of the at least one firstobject; a three-dimensional (“3D”) image or hologram; or an image of theat least one first object superimposed over a targeted portion of thework environment indicating at least one of position, orientation, orconfiguration of the at least one first object intended for placement atthe targeted portion; and/or the like. In some cases, thehighlighting-based instruction might include, but is not limited to, atleast one of a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over the at least onefirst object; a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a fourth object thatis related to the first task associated with the at least one firstobject; or a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a targeted portionof the work environment indicated intended placement of the at least onefirst object at the targeted portion; and/or the like.

In some embodiments, the first task might include, without limitation,sorting, categorizing, arranging, or organizing the one or more firstobjects, wherein the one or more objects might include, but is notlimited to, one or more microscope slides each containing a test sampleor a tissue sample. In some cases, a plurality of users are tasked withperforming the first task, wherein each user wears an AR headset thatdisplays generated image overlays to each user that distinguishesobjects that one user is intended to interact with from objects that theother users among the plurality of users are intended to interact withas part of the first task. Alternatively, the first task might include,without limitation, one of running a test sample or a tissue sampleusing laboratory instrumentation; assembling a piece of machinery;assembling an object; preparing a test sample or a tissue sample;identifying or tracking samples within a laboratory; instructing orguiding users regarding how to perform experimental protocols or testsin a laboratory; performing processes in a histology laboratory;performing processes in a pathologist's office; performing tasks afterslide diagnosis; performing processes in a clinical or analyticallaboratory; or transferring reagents or samples; and/or the like. Insome cases, the one or more first objects might include, but is notlimited to, one of laboratory instrumentation, laboratory tools, sampletransfer devices, puzzle components or pieces, machinery parts, assemblytools, measurement tools, object parts, sample reagents, samplecontainers, burners, coolers, mixers, sample preparation tools, orsample transfer tools, and/or the like. In some cases, the workenvironment comprises one of a laboratory, a construction site, amachine shop, a workshop, a factory, or a room, and/or the like.

In accordance with the various embodiments described herein, theaugmented reality (“AR”) system provides for generation of imageoverlays (including, but not limited to, at least one of graphicalicon-based overlays, text-based overlays, image-based overlays, orhighlighting-based overlays, and/or the like) that are superimposed overeither an actual view of an area (e.g., work environment or the like) oran image or video of the area. In the former case, the actual view ofthe area might be viewed by the user through a lens, visor, transparentscreen, or the like, with the generated image overlays either beingprojected directly into the eyes of the user, being projected on aprojection plane or surface in front of the eyes of the user (e.g., onthe plane or surface defined by the lens, visor, transparent screen, orthe like), or being displayed on the surface of the lens, visor, ortransparent screen, or the like. Alternatively, for AR headsets withouta lens, visor, transparent screen, or other component disposed betweenthe eyes of the user (or the user's personal corrective lenses), theactual view of the area might be viewed directly by the eyes of the user(or through only the user's personal corrective lenses), with thegenerated image overlays either being projected directly into the eyesof the user or being projected as a hologram in front of the eyes of theuser, or the like. In the latter case, the image or video of the areamight be viewed by the user using a display screen disposed in front ofthe user, with the generated image overlays being superimposed over theimage or video of the area prior to the composite image or video beingdisplayed on the display screen. The use of AR headsets providesportability and flexibility within the work environment, among otheradvantages that overcome the issues inherent with the conventionaltechniques (such as the conventional hand sorting scenarios discussedabove).

These and other aspects of the AR-based assistance system andfunctionality are described in greater detail with respect to thefigures. In some aspects, the AR-based assistance system andfunctionality may also be integrated with a smart lab, contentmanagement, or lab workflow management systems, such as, but not limitedto, Agilent OpenLab® or other similar software suite, or the like.

The following detailed description illustrates a few exemplaryembodiments in further detail to enable one of skill in the art topractice such embodiments. The described examples are provided forillustrative purposes and are not intended to limit the scope of theinvention.

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the described embodiments. It will be apparent to oneskilled in the art, however, that other embodiments of the presentinvention may be practiced without some of these specific details. Inother instances, certain structures and devices are shown in blockdiagram form. Several embodiments are described herein, and whilevarious features are ascribed to different embodiments, it should beappreciated that the features described with respect to one embodimentmay be incorporated with other embodiments as well. By the same token,however, no single feature or features of any described embodimentshould be considered essential to every embodiment of the invention, asother embodiments of the invention may omit such features.

Unless otherwise indicated, all numbers used herein to expressquantities, dimensions, and so forth used should be understood as beingmodified in all instances by the term “about.” In this application, theuse of the singular includes the plural unless specifically statedotherwise, and use of the terms “and” and “or” means “and/or” unlessotherwise indicated. Moreover, the use of the term “including,” as wellas other forms, such as “includes” and “included,” should be considerednon-exclusive. Also, terms such as “element” or “component” encompassboth elements and components comprising one unit and elements andcomponents that comprise more than one unit, unless specifically statedotherwise.

Various embodiments described herein, while embodying (in some cases)software products, computer-performed methods, and/or computer systems,represent tangible, concrete improvements to existing technologicalareas, including, without limitation, user assistance technology, and/orthe like. In other aspects, certain embodiments, can improve thefunctioning of user equipment or systems themselves (e.g., userassistance systems, computing systems, etc.), for example, by receiving,with a computing system and from one or more first image capture deviceshaving a field of view of at least a first portion of a workenvironment, one or more first images of the at least a first portion ofthe work environment, the one or more first images overlapping with afield of view of eyes of a first user wearing an augmented reality(“AR”) headset; analyzing, with the computing system, the received oneor more first images to identify one or more first objects disposed inthe work environment; querying at least one database, with the computingsystem, to determine a first task associated with at least one firstobject among the identified one or more first objects; generating, withthe computing system, a first image overlay, the first image overlayproviding one or more instructions (which might include, withoutlimitation, at least one of a graphical icon-based instruction, atext-based instruction, an image-based instruction, or ahighlighting-based instruction, and/or the like) to be presented to auser to implement the first task associated with the at least one firstobject; and displaying, with the computing system and to the eyes of thefirst user through the AR headset, the generated first image overlaythat overlaps with the field of view of the eyes of the first user;and/or the like. In particular, to the extent any abstract concepts arepresent in the various embodiments, those concepts can be implemented asdescribed herein by devices, software, systems, and methods that involvespecific novel functionality (e.g., steps or operations), such as,generating, with the augmented reality (“AR”) system, image overlays(including, but not limited to, at least one of graphical icon-basedoverlays, text-based overlays, image-based overlays, orhighlighting-based overlays, and/or the like) that are superimposed overeither an actual view of an area (e.g., work environment or the like) oran image or video of the area, the generated image overlays providing atleast one of a graphical icon-based instruction, a text-basedinstruction, an image-based instruction, or a highlighting-basedinstruction each indicative of one or more instructions to be presentedto a user to implement the first task associated with the at least onefirst object, and/or the like, to name a few examples, that extendbeyond mere conventional computer processing operations. Thesefunctionalities can produce tangible results outside of the implementingcomputer system, including, merely by way of example, optimizedpresentation of instructions within an AR environment to assist a userin the performance of tasks within a work environment, and/or the like,at least some of which may be observed or measured by customers and/orservice providers.

In an aspect, a method might comprise receiving, with a computing systemand from one or more first image capture devices having a field of viewof at least a first portion of a work environment, one or more firstimages of the at least a first portion of the work environment, the oneor more first images overlapping with a field of view of eyes of a firstuser wearing an augmented reality (“AR”) headset; analyzing, with thecomputing system, the received one or more first images to identify oneor more first objects disposed in the work environment; and querying atleast one database, with the computing system, to determine a first taskassociated with at least one first object among the identified one ormore first objects, wherein the first task comprises sorting,categorizing, arranging, or organizing the one or more first objects,wherein the one or more objects comprise one or more microscope slideseach containing a test sample or a tissue sample. The method mightfurther comprise generating, with the computing system, a first imageoverlay, the first image overlay providing at least one of a graphicalicon-based instruction, a text-based instruction, an image-basedinstruction, or a highlighting-based instruction each indicative of oneor more instructions to be presented to a user to implement the sorting,categorizing, arranging, or organizing of the one or more microscopeslides containing test or tissue samples; and displaying, with thecomputing system and to the eyes of the first user through the ARheadset, the generated first image overlay that overlaps with the fieldof view of the eyes of the first user.

In another aspect, a method might comprise receiving, with a computingsystem and from one or more first image capture devices having a fieldof view of at least a first portion of a work environment, one or morefirst images of the at least a first portion of the work environment,the one or more first images overlapping with a field of view of eyes ofa first user wearing an augmented reality (“AR”) headset; analyzing,with the computing system, the received one or more first images toidentify one or more first objects disposed in the work environment; andquerying at least one database, with the computing system, to determinea first task associated with at least one first object among theidentified one or more first objects. The method might further comprisegenerating, with the computing system, a first image overlay, the firstimage overlay providing one or more instructions to be presented to auser to implement the first task associated with the at least one firstobject; and displaying, with the computing system and to the eyes of thefirst user through the AR headset, the generated first image overlaythat overlaps with the field of view of the eyes of the first user.

In some embodiments, the computing system might comprise one of aprocessor disposed in the AR headset, a computing system disposed in thework environment, a remote computing system disposed external to thework environment and accessible over a network, or a cloud computingsystem, and/or the like. In some cases, the AR headset might compriseone of a set of AR goggles, a pair of AR-enabled eyewear, an AR-enabledsmartphone mounted in a headset, or an AR helmet, and/or the like.

According to some embodiments, the AR headset might comprise one or moreof at least one earpiece or at least one speaker, where the method mightfurther comprise: generating, with the computing system, one or moreaudio-based instructions corresponding to each of at least one of theone or more instructions to be presented to the user via the generatedfirst image overlay; and presenting, with the computing system and viaone of the one or more of the at least one earpiece or the at least onespeaker, the generated one or more audio-based instructions inconjunction with displaying corresponding each of at least one of theone or more instructions being presented to the user via the generatedfirst image overlay as displayed to the eyes of the first user throughthe AR headset.

In some cases, displaying the generated first image overlay to the eyesof the first user through the AR headset comprises one of projecting thegenerated first image overlay directly in the eyes of the first user,projecting the generated first image overlay on a projection plane orsurface in front of the eyes of the first user, projecting the generatedfirst image overlay as a hologram in front of the eyes of the firstuser, displaying the generated first image overlay on a transparent orsemi-transparent display screen of the AR headset that is disposed infront of the eyes of the first user, or displaying the generated firstimage overlay superimposed over a continuous video recording and displayon a display screen of an AR-enabled smartphone mounted in the ARheadset, and/or the like.

In some instances, at least one of the one or more first image capturedevices might be disposed on a portion of a housing of the AR headset.According to some embodiments, the method might further comprisereceiving, with the computing system and from one or more second imagecapture devices that are disposed within the work environment butexternal to the AR headset, one or more second images of at least asecond portion of the work environment. In such cases, analyzing, withthe computing system, the received one or more first images to identifyone or more first objects disposed in the work environment mightcomprise analyzing, with the computing system, the received one or morefirst images and the received one or more second images to identify theone or more first objects disposed in the work environment.

In some embodiments, at least one of the one or more instructions, whendisplayed within the generated first image overlay, is superimposedover, displayed around, or displayed beside the at least one firstobject as viewed by the user through the AR headset. In some instances,the at least one of the one or more instructions might comprise at leastone of a graphical icon-based instruction, a text-based instruction, animage-based instruction, or a highlighting-based instruction, and/or thelike. In some cases, the graphical icon-based instruction might compriseat least one of a graphical icon representing identification informationassociated with the at least one first object, a graphical iconrepresenting identification information associated with a second objectwith which the at least one first object is intended to interact as partof the first task, a colored graphical icon distinguishing one firstobject from another first object among the at least one first object, agraphical icon comprising a directional arrow representing a directionthat the at least one first object should be taken as part of the firsttask, or a colored graphical icon comprising a colored directional arrowdistinguishing a first direction that the at least one first objectshould be taken as part of the first task from a second direction that athird object should be taken as part of the first task, and/or the like.In some instances, the text-based instruction might comprise at leastone of a floating text window comprising textual instructionscorresponding to the first task or a surface text window that issuperimposed on one of a table-top surface, a wall surface, or an objectsurface and that comprises textual instructions corresponding to thefirst task, wherein the floating text window, when displayed within thegenerated first image overlay, is displayed as a floating image besidethe at least one first object or displayed as a floating image withinthe field of view of the eyes of the first user. In some cases, theimage-based instruction might comprise at least one of an imagerepresenting identification information associated with the at least onefirst object, an image representing identification informationassociated with a second object with which the at least one first objectis intended to interact as part of the first task, an image comprising adirectional arrow representing a direction that the at least one firstobject should be taken as part of the first task, an image comprisingimages of numbers or codes representing an order of processes of thefirst task associated with the at least one first object, a magnifiedimage of the at least one first object, a three-dimensional (“3D”) imageor hologram, or an image of the at least one first object superimposedover a targeted portion of the work environment indicating at least oneof position, orientation, or configuration of the at least one firstobject intended for placement at the targeted portion, and/or the like.In some instances, the highlighting-based instruction might comprise atleast one of a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over the at least onefirst object, a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a fourth object thatis related to the first task associated with the at least one firstobject, or a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a targeted portionof the work environment indicated intended placement of the at least onefirst object at the targeted portion, and/or the like.

According to some embodiments, the one or more first objects each mightcomprise a unique identifier, wherein the unique identifier comprises aunique numerical identifier, a unique alphanumeric identifier, a uniquesymbolic identifier, a unique one-dimensional (“1D”) visual code(including, but not limited to, a unique bar code, or other unique 1Dpattern code, or the like), a unique two-dimensional (“2D”) visual code(including, but not limited to, a unique quick response (“QR”) code, orother unique 2D pattern or matrix code, or the like), a uniquethree-dimensional (“3D”) visual code (including, but not limited to, aunique 3D pattern code, a unique 3D matrix code, a unique 3D holographiccode, a unique 3D holographic pattern code, a unique 3D holographicmatrix code, or the like), a digital image of at least a portion of thefirst object itself or of a representative example of the first object(e.g., digital image of at least a portion of a tissue sample or tissuesection, table, tray, container, instrumentation, tool, or other object,etc.), or a unique identifying feature or attribute (of a first objectamong the one or more first objects, or the like), and/or the like,wherein analyzing, with the computing system, the received one or morefirst images to identify one or more first objects disposed in the workenvironment might comprise identifying, with the computing system, aunique identifier that is associated with each of the one or more firstobjects and that is displayed on at least a portion of each of the oneor more first objects. Merely by way of example, in some cases, the oneor more first objects each might comprise a radio frequency identifier(“RFID”) tag affixed thereto, wherein identifying the one or more firstobjects might comprise identifying the one or more first objects basedon the RFID tag affixed to each first object in conjunction withidentifying the one or more first objects based on analysis of thereceived one or more first images.

In some instances, the method might further comprise illuminating, witha first light source, one or more of the at least one first object orthe at least a first portion of the work environment during at least oneof the following periods: prior to the one or more first images beingcaptured by the one or more first image capture device; or inconjunction with the display of the generated first image overlay. Insome cases, the first light source might be disposed on a portion of theAR headset.

In some embodiments, the method might further comprise tracking, withthe computing system and using one or more second image capture devicesthat are disposed on the AR headset and that have a field of view thatincludes the eyes of the first user, the eyes of the first user todetermine a direction in which the first user is looking; anddetermining, with the computing system, a central portion of a field ofview of the first user based at least in part on the determineddirection that the first user is looking in conjunction with the one ormore first images as captured by the one or more first image capturedevices. In some instances, the method might further comprise, inresponse to determining the central portion of the field of view of thefirst user, identifying, with the computing system, at least one fifthobject.

According to some embodiments, the first task might comprise sorting,categorizing, arranging, or organizing the one or more first objects,wherein the one or more objects might comprise one or more microscopeslides each containing a test sample or a tissue sample. Alternatively,the first task might comprise one of running a test sample or a tissuesample using laboratory instrumentation, solving a puzzle, assembling apiece of machinery, assembling an object, preparing a test sample or atissue sample, identifying or tracking samples within a laboratory,instructing or guiding users regarding how to perform experimentalprotocols or tests in a laboratory, performing processes in a histologylaboratory, performing processes in a pathologist's office, performingtasks after slide diagnosis, performing processes in a clinical oranalytical laboratory, or transferring reagents or samples, and/or thelike, wherein the one or more first objects might comprise one oflaboratory instrumentation, laboratory tools, sample transfer devices,puzzle components or pieces, machinery parts, assembly tools,measurement tools, object parts, sample reagents, sample containers,burners, coolers, mixers, sample preparation tools, or sample transfertools, and/or the like. In some cases, a plurality of users may betasked with performing the first task, wherein each user might wear anAR headset that displays generated image overlays to each user thatdistinguishes objects that one user is intended to interact with fromobjects that the other users among the plurality of users are intendedto interact with as part of the first task. In some instances, the workenvironment might comprise one of a laboratory, a construction site, amachine shop, a workshop, a factory, or a room, and/or the like.

In yet another aspect, a system might comprise a computing system, whichmight comprise at least one first processor and a first non-transitorycomputer readable medium communicatively coupled to the at least onefirst processor. The first non-transitory computer readable medium mighthave stored thereon computer software comprising a first set ofinstructions that, when executed by the at least one first processor,causes the computing system to: receive, from one or more first imagecapture devices having a field of view of at least a first portion of awork environment, one or more first images of the at least a firstportion of the work environment, the one or more first imagesoverlapping with a field of view of eyes of a first user wearing anaugmented reality (“AR”) headset; analyze the received one or more firstimages to identify one or more first objects disposed in the workenvironment; query at least one database to determine a first taskassociated with at least one first object among the identified one ormore first objects; generate a first image overlay, the first imageoverlay providing one or more instructions to be presented to a user toimplement the first task associated with the at least one first object;and display, to the eyes of the first user through the AR headset, thegenerated first image overlay that overlaps with the field of view ofthe eyes of the first user.

In some embodiments, the computing system might comprise one of aprocessor disposed in the AR headset, a computing system disposed in thework environment, a remote computing system disposed external to thework environment and accessible over a network, or a cloud computingsystem, and/or the like. In some cases, the AR headset might compriseone of a set of AR goggles, a pair of AR-enabled eyewear, an AR-enabledsmartphone mounted in a headset, or an AR helmet, and/or the like.

According to some embodiments, the AR headset might comprise one or moreof at least one earpiece or at least one speaker, where the first set ofinstructions, when executed by the at least one first processor, mightfurther cause the computing system to: generate one or more audio-basedinstructions corresponding to each of at least one of the one or moreinstructions to be presented to the user via the generated first imageoverlay; and present, via one of the one or more of the at least oneearpiece or the at least one speaker, the generated one or moreaudio-based instructions in conjunction with displaying correspondingeach of at least one of the one or more instructions being presented tothe user via the generated first image overlay as displayed to the eyesof the first user through the AR headset.

In some cases, displaying the generated first image overlay to the eyesof the first user through the AR headset comprises one of projecting thegenerated first image overlay directly in the eyes of the first user,projecting the generated first image overlay on a projection plane orsurface in front of the eyes of the first user, projecting the generatedfirst image overlay as a hologram in front of the eyes of the firstuser, displaying the generated first image overlay on a transparent orsemi-transparent display screen of the AR headset that is disposed infront of the eyes of the first user, or displaying the generated firstimage overlay superimposed over a continuous video recording and displayon a display screen of an AR-enabled smartphone mounted in the ARheadset, and/or the like.

In some instances, at least one of the one or more first image capturedevices might be disposed on a portion of a housing of the AR headset.According to some embodiments, the first set of instructions, whenexecuted by the at least one first processor, might further cause thecomputing system to: receive, from one or more second image capturedevices that are disposed within the work environment but external tothe AR headset, one or more second images of at least a second portionof the work environment. In such cases, analyzing the received one ormore first images to identify one or more first objects disposed in thework environment might comprise analyzing the received one or more firstimages and the received one or more second images to identify the one ormore first objects disposed in the work environment.

In some embodiments, at least one of the one or more instructions, whendisplayed within the generated first image overlay, is superimposedover, displayed around, or displayed beside the at least one firstobject as viewed by the user through the AR headset. In some instances,the at least one of the one or more instructions might comprise at leastone of a graphical icon-based instruction, a text-based instruction, animage-based instruction, or a highlighting-based instruction, and/or thelike. In some cases, the graphical icon-based instruction might compriseat least one of a graphical icon representing identification informationassociated with the at least one first object, a graphical iconrepresenting identification information associated with a second objectwith which the at least one first object is intended to interact as partof the first task, a colored graphical icon distinguishing one firstobject from another first object among the at least one first object, agraphical icon comprising a directional arrow representing a directionthat the at least one first object should be taken as part of the firsttask, or a colored graphical icon comprising a colored directional arrowdistinguishing a first direction that the at least one first objectshould be taken as part of the first task from a second direction that athird object should be taken as part of the first task, and/or the like.In some instances, the text-based instruction might comprise at leastone of a floating text window comprising textual instructionscorresponding to the first task or a surface text window that issuperimposed on one of a table-top surface, a wall surface, or an objectsurface and that comprises textual instructions corresponding to thefirst task, wherein the floating text window, when displayed within thegenerated first image overlay, is displayed as a floating image besidethe at least one first object or displayed as a floating image withinthe field of view of the eyes of the first user. In some cases, theimage-based instruction might comprise at least one of an imagerepresenting identification information associated with the at least onefirst object, an image representing identification informationassociated with a second object with which the at least one first objectis intended to interact as part of the first task, an image comprising adirectional arrow representing a direction that the at least one firstobject should be taken as part of the first task, an image comprisingimages of numbers or codes representing an order of processes of thefirst task associated with the at least one first object, a magnifiedimage of the at least one first object, a three-dimensional (“3D”) imageor hologram, or an image of the at least one first object superimposedover a targeted portion of the work environment indicating at least oneof position, orientation, or configuration of the at least one firstobject intended for placement at the targeted portion, and/or the like.In some instances, the highlighting-based instruction might comprise atleast one of a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over the at least onefirst object, a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a fourth object thatis related to the first task associated with the at least one firstobject, or a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a targeted portionof the work environment indicated intended placement of the at least onefirst object at the targeted portion, and/or the like.

According to some embodiments, the one or more first objects each mightcomprise a unique identifier, wherein the unique identifier comprises aunique numerical identifier, a unique alphanumeric identifier, a uniquesymbolic identifier, a unique one-dimensional (“1D”) visual code(including, but not limited to, a unique bar code, or other unique 1Dpattern code, or the like), a unique two-dimensional (“2D”) visual code(including, but not limited to, a unique quick response (“QR”) code, orother unique 2D pattern or matrix code, or the like), a uniquethree-dimensional (“3D”) visual code (including, but not limited to, aunique 3D pattern code, a unique 3D matrix code, a unique 3D holographiccode, a unique 3D holographic pattern code, a unique 3D holographicmatrix code, or the like), a digital image of at least a portion of thefirst object itself or of a representative example of the first object(e.g., digital image of at least a portion of a tissue sample or tissuesection, table, tray, container, instrumentation, tool, or other object,etc.), or a unique identifying feature or attribute (of a first objectamong the one or more first objects, or the like), and/or the like,wherein analyzing the received one or more first images to identify oneor more first objects disposed in the work environment might compriseidentifying a unique identifier that is associated with each of the oneor more first objects and that is displayed on at least a portion ofeach of the one or more first objects. Merely by way of example, in somecases, the one or more first objects each might comprise a radiofrequency identifier (“RFID”) tag affixed thereto, wherein identifyingthe one or more first objects might comprise identifying the one or morefirst objects based on the RFID tag affixed to each first object inconjunction with identifying the one or more first objects based onanalysis of the received one or more first images.

In some instances, the system might further comprise a first lightsource, which might illuminate one or more of the at least one firstobject or the at least a first portion of the work environment during atleast one of the following periods: prior to the one or more firstimages being captured by the one or more first image capture device; orin conjunction with the display of the generated first image overlay. Insome cases, the first light source might be disposed on a portion of theAR headset.

In some embodiments, the first set of instructions, when executed by theat least one first processor, might further cause the computing systemto: track, using one or more second image capture devices that aredisposed on the AR headset and that have a field of view that includesthe eyes of the first user, the eyes of the first user to determine adirection in which the first user is looking; and determine a centralportion of a field of view of the first user based at least in part onthe determined direction that the first user is looking in conjunctionwith the one or more first images as captured by the one or more firstimage capture devices. In some instances, the first set of instructions,when executed by the at least one first processor, might further causethe computing system to: in response to determining the central portionof the field of view of the first user, identify at least one fifthobject.

According to some embodiments, the first task might comprise sorting,categorizing, arranging, or organizing the one or more first objects,wherein the one or more objects might comprise one or more microscopeslides each containing a test sample or a tissue sample. Alternatively,the first task might comprise one of running a test sample or a tissuesample using laboratory instrumentation, solving a puzzle, assembling apiece of machinery, assembling an object, preparing a test sample or atissue sample, identifying or tracking samples within a laboratory,instructing or guiding users regarding how to perform experimentalprotocols or tests in a laboratory, performing processes in a histologylaboratory, performing processes in a pathologist's office, performingtasks after slide diagnosis, performing processes in a clinical oranalytical laboratory, or transferring reagents or samples, and/or thelike, wherein the one or more first objects might comprise one oflaboratory instrumentation, laboratory tools, sample transfer devices,puzzle components or pieces, machinery parts, assembly tools,measurement tools, object parts, sample reagents, sample containers,burners, coolers, mixers, sample preparation tools, or sample transfertools, and/or the like. In some cases, a plurality of users may betasked with performing the first task, wherein each user might wear anAR headset that displays generated image overlays to each user thatdistinguishes objects that one user is intended to interact with fromobjects that the other users among the plurality of users are intendedto interact with as part of the first task. In some instances, the workenvironment might comprise one of a laboratory, a construction site, amachine shop, a workshop, a factory, or a room, and/or the like.

In still another aspect, an apparatus might comprise at least oneprocessor and a non-transitory computer readable medium communicativelycoupled to the at least one processor. The non-transitory computerreadable medium might have stored thereon computer software comprising aset of instructions that, when executed by the at least one processor,causes the apparatus to: receive, from one or more first image capturedevices having a field of view of at least a first portion of a workenvironment, one or more first images of the at least a first portion ofthe work environment, the one or more first images overlapping with afield of view of eyes of a first user wearing an augmented reality(“AR”) headset; analyze the received one or more first images toidentify one or more first objects disposed in the work environment;query at least one database to determine a first task associated with atleast one first object among the identified one or more first objects;generate a first image overlay, the first image overlay providing one ormore instructions to be presented to a user to implement the first taskassociated with the at least one first object; and display, to the eyesof the first user through the AR headset, the generated first imageoverlay that overlaps with the field of view of the eyes of the firstuser.

Various modifications and additions can be made to the embodimentsdiscussed without departing from the scope of the invention. Forexample, while the embodiments described above refer to particularfeatures, the scope of this invention also includes embodiments havingdifferent combination of features and embodiments that do not includeall of the above described features.

Specific Exemplary Embodiments

We now turn to the embodiments as illustrated by the drawings. FIGS. 1-6illustrate some of the features of the method, system, and apparatus forimplementing assistance within a work environment, and, moreparticularly, to methods, systems, and apparatuses for implementingaugmented reality (AR)-based assistance within a work environment, asreferred to above. The methods, systems, and apparatuses illustrated byFIGS. 1-6 refer to examples of different embodiments that includevarious components and steps, which can be considered alternatives orwhich can be used in conjunction with one another in the variousembodiments. The description of the illustrated methods, systems, andapparatuses shown in FIGS. 1-6 is provided for purposes of illustrationand should not be considered to limit the scope of the differentembodiments.

With reference to the figures, FIG. 1 is a schematic diagramillustrating a system 100 for implementing augmented reality (AR)-basedassistance within a work environment, in accordance with variousembodiments.

In the non-limiting embodiment of FIG. 1, system 100 might comprisecomputing system 105 a and corresponding database(s) 110 a. In someinstances, the database(s) 110 a might be local to the computing system105 a, in some cases, integrated within the computing system 105 a. Inother cases, the database 110 a might be external, yet communicativelycoupled, to the computing system 105 a. System 100, according to someembodiments, might further comprise one or more augmented reality (“AR”)headsets 115 a-115 n (collectively, “AR headsets 115” or the like) thatare worn or wearable by one or more users 120 a-120 n (collectively,“users 120” or the like). In some cases, the AR headset might include,but is not limited to, one of a set of AR goggles, a pair of AR-enabledeyewear, an AR-enabled smartphone mounted in a headset, or an AR helmet,and/or the like. In some embodiments, system 100 might comprise one ormore objects 125 a-125 n (collectively, “objects 125” or the like); insome cases, objects 125 might comprise identifier or identification(“ID”) tags 130 (optional) affixed thereto (the ID tags 130 comprisingtags 130 a-130 n, or the like, each affixed to one of the objects 125).In some instances, the ID tags 130 might comprise a unique identifier,which might include, without limitation, a unique numerical identifier,a unique alphanumeric identifier, a unique symbolic identifier, a uniqueone-dimensional (“1D”) visual code (including, but not limited to, aunique bar code, or other unique 1D pattern code, or the like), a uniquetwo-dimensional (“2D”) visual code (including, but not limited to, aunique quick response (“QR”) code, or other unique 2D pattern or matrixcode, or the like), a unique three-dimensional (“3D”) visual code(including, but not limited to, a unique 3D pattern code, a unique 3Dmatrix code, a unique 3D holographic code, a unique 3D holographicpattern code, a unique 3D holographic matrix code, or the like), adigital image of at least a portion of the object 125 itself or of arepresentative example of the object 125 (e.g., digital image of atleast a portion of a tissue sample or tissue section, table, tray,container, instrumentation, tool, or other object, etc.), or a uniqueidentifying feature or attribute (of an object among the one or moreobjects 125, or the like), and/or the like. Alternatively, oradditionally, at least one of the ID tags 130 might comprise a radiofrequency identifier (“RFID”) tag, or the like, that when interrogatedby a radio frequency scanner (which may be disposed in the AR headset115, such as transceiver 190 or other component (not shown), or may bedisposed within a dedicated RFID scanner or other similar device (notshown)) would broadcast identification information (which might includethe unique identifier data, or the like) in rf response signals. [Thisdescribes a passive RFID tag, although a battery-assisted passive RFIDtag may be used. An active RFID tag that broadcasts its data withoutneed of power from an interrogating signal from a rf scanner (in somecases, broadcasting constantly) could be used for certain tasks.] Insome embodiments, a smart tag that combines RFID tag functionality withprinted ID functionality (e.g., with use of barcodes or other 1D visualcodes, QR codes or other 2D visual codes, 3D visual codes, numbers,letters, text, code, etc.) may also be used. Alternative, or additional,to the ID tags 130, image recognition or photogrammetric recognitionfunctionality (particularly as enhanced by use of AI system 140 or thelike) may be used to perform, or aid in the performance of,identification of objects of interest among the objects 125. Forexample, the system might capture a current digital image of at least aportion of an object 125 and might utilize image recognition orphotogrammetric recognition functionality to identify the object 125based on analysis of the captured current digital image of the at leasta portion of the object 125 itself (i.e., by identifying a uniqueidentifying feature or attribute of the object 125, or the like).Alternatively, or additionally, the system might utilize imagerecognition or photogrammetric recognition functionality to identify theobject 125 based on analysis of a digital image of at least a portion ofthe object 125 itself or of a representative example of the object 125that is used as an ID tag 130 on the object 125. The computing system105 a, the database(s) 110 a, the AR goggles 115 a-115 n that are wornor wearable by respective users 120 a-120 n, and the objects 125 a-125n, or the like, might be disposed in work environment 135, which mightinclude, but is not limited to, one of a laboratory, a constructionsite, a machine shop, a workshop, a factory, or a room, and/or the like.

System 100 might further comprise an artificial intelligence (“AI”)system 140 (optional) that might communicatively couple to computingsystem 105 a via network(s) 145. In some embodiments, the AI system140—which might include, but is not limited to, at least one of amachine learning system(s), a learning algorithm-based system(s), or aneural network system(s), and/or the like—might be used to improveidentification of objects by learning how to optimize resolution ofimages of visual-based ID tags of objects or by learning how toimplement and optimize image processing techniques (including, withoutlimitation, panning, tilting, and/or zooming, or the like) to orientand/or magnify images of visual-based ID tags, and/or the like. The AIsystem 140 might also be used to learn or anticipate identification ofobjects (e.g., by improving image recognition or photogrammetricrecognition of objects, or the like), to learn or anticipateidentification of tasks (e.g., to improve communication with databases,or to improve query techniques used to query the databases, or thelike), to learn or anticipate behavior of users (e.g., to anticipate orfacilitate commands by the users, etc.), or the like. In some cases,feedback loops of data may be used as part of the learning processesimplemented by the AI system 140. According to some embodiments,alternative or additional to the computing system 105 a andcorresponding database 110 a being disposed within work environment 135,system 100 might comprise remote computing system 105 b (optional) andcorresponding database(s) 110 b (optional) that communicatively couplewith at least one of the one or more AR headsets 115 a-115 n in the workenvironment 135 via the one or more networks 145. Herein, although somecomponents of system 100 are indicated as being optional while othersare not, this is merely for the particular embodiment as shown, and, inother embodiments, one or more of the former set of components (orcomponents indicated as being “optional”) may be required while one ormore of the latter set of components (or components not indicated asbeing “optional”) may in fact be optional.

According to some embodiments, computing system 105 a might include,without limitation, one of a processor disposed in the AR headset, or acomputing system disposed in the work environment, and/or the like. Insome cases, remote computing system 105 b might include, but is notlimited to, at least one of a remote computing system disposed externalto the work environment and accessible over a network, or a cloudcomputing system, and/or the like. The AI system 140 might be used tofacilitate operation by computing system 105 a, computing system 105 b,and/or at least one AR headset 115. Merely by way of example, network(s)145 might each include a local area network (“LAN”), including, withoutlimitation, a fiber network, an Ethernet network, a Token-Ring™ network,and/or the like; a wide-area network (“WAN”); a wireless wide areanetwork (“WWAN”); a virtual network, such as a virtual private network(“VPN”); the Internet; an intranet; an extranet; a public switchedtelephone network (“PSTN”); an infra-red network; a wireless network,including, without limitation, a network operating under any of the IEEE802.11 suite of protocols, the Bluetooth™ protocol known in the art,and/or any other wireless protocol; and/or any combination of theseand/or other networks. In a particular embodiment, network(s) 145 mighteach might include an access network of an Internet service provider(“ISP”). In another embodiment, network(s) 145 might each might includea core network of the ISP, and/or the Internet.

In some embodiments, each AR headset (e.g., AR headset 115 n as shown inFIG. 1, or the like) might include, without limitation, at least one ofa processor 150, a data store 155, a speaker(s) or earpiece(s) 160(optional), eye tracking sensor(s) 165 (optional), light source(s) 170(optional), audio sensor(s) or microphone(s) 175 (optional), front orfront-facing camera(s) 180, display screen(s) or projector(s) 185,and/or transceiver(s) 190, and/or the like. In some instances, theprocessor 150 might communicatively couple (e.g., via a bus, via wiredconnectors, or via electrical pathways (e.g., traces and/or pads, etc.)of printed circuit boards (“PCBs”) or integrated circuits (“ICs”),and/or the like) to each of one or more of the data store 155, thespeaker(s) or earpiece(s) 160 (optional), the eye tracking sensor(s) 165(optional), the light source(s) 170 (optional), the audio sensor(s) ormicrophone(s) 175 (optional), the front camera(s) 180, the displayscreen(s) or projector(s) 185, and/or the transceiver(s) 190, and/or thelike.

The processor 150 might perform functionalities of the AR-basedassistance as described herein, either as a separate computing system orin conjunction with the computing system 105 a or 105 b and/or with theAI system 140. The data store 155 (or the database(s) 110 a or 110 b)might store at least one of data regarding identified objects, dataregarding tasks, data regarding the first task (and processes associatedtherewith), data regarding user preferences of each of one or moreparticular users, data regarding communications protocols or identifierswith other devices (e.g., computing system 105 a or 105 b, AI system140, or the like) or user devices (not shown; including, but not limitedto, smart phones, mobile phones, tablet computers, laptop computers,desktop computers, or monitors, or the like), and/or the like. Thespeaker(s) (which might be disposed on a housing of the AR headset 115and directed toward the ears of the user or directed around the ARheadset 115) or earpiece(s) 160 (which are intended to fit in or aroundthe user's ears, or the like) might provide aural or verbal informationor instructions to the user. The eye tracking sensor(s) 165—which mightinclude, without limitation, at least one of one or more cameras, one ormore motion sensors, or one or more tracking sensors, and/or thelike—track where the user's eyes are looking, which in conjunction withcomputation processing by the processor 150, the computing system 105 aor 105 b, and/or the AI system 140 to compare with images or videostaken in front of the AR headset 115, to determine what objects orportions of the work environment 135 the user is looking at or focusingon. The light source(s) 170 might provide at least one of broadillumination of a portion of the work environment, illumination ofobjects of interest as an alternative to image overlays highlightingsuch objects of interest, or illumination of a portion of the workenvironment corresponding to what the user is looking at, and/or thelike. The audio sensor(s) 175 might include, but is not limited to,microphones, sound sensors, or noise sensors, and/or the like, and mightbe used to receive or capture voice signals, sound signals, and/or noisesignals, or the like that may be processed to detect input, commands, orinteractions by the user, to detect sounds relevant to tasks beingperformed within the work environment 135, or to detect noise signalsthat may be used to improve noise cancelling functionality of the ARheadset 115, and/or the like. The front camera(s) 180 may be used tocapture images or video of an area in front of the AR headset 115, witha field of view of each front camera 180 overlapping with a field ofview of an eye(s) of the user 120, the captured images or video. Thedisplay screen(s) and/or projector(s) 185 may be used to display orproject the generated image overlays (and/or to display a compositeimage or video that combines the generated image overlays superimposedover images or video of the actual area). The transceiver 190 is used tocommunicate wirelessly with the computing system 105 a or with acommunications relay device(s) (not shown), to transfer data (including,but not limited to, at least one of data regarding identified objects,data regarding tasks, data regarding the first task (and processesassociated therewith), data regarding user preferences of each of one ormore particular users, data regarding communications protocols oridentifiers with other devices, and/or the like. In some embodiments,wireless AR headsets may be used (as depicted in FIG. 1 by lightningbolt symbols between AR headsets 115 and computing system 105 a), whilewired AR headsets may be in used in other embodiments (not shown in FIG.1).

According to some embodiments, system 100 might further comprise one ormore (area) cameras 195 (optional) that are disposed within workenvironment 135 yet external to any of the AR headsets 115, the one ormore cameras 195 being communicatively coupled to the computing system105 a via wired communication (depicted in FIG. 1 by the line betweenthe camera(s) 195 and the computing system 105 a) or via wirelesscommunication (depicted in FIG. 1 by the lightning bolt symbol betweenthe camera(s) 195 and the computing system 105 a). These cameras 195 maybe used to capture images or videos of areas within the work environment135 in other to supplement and/or complement the images or videoscaptured by the front camera(s) 180 housed on each of the AR headsets115. In this manner, the images or video captured by the camera(s) 180of an AR headset 115 together with the images or video captured by thecamera(s) 195 may be analyzed by the processor 150, the computing system105 a or 105 b, and/or the AI system 140 to identify objects of interestamong the objects 125, in some cases, providing different fields of viewor visual perspectives to capture more easily identifiable features ofthe objects of interest or to capture more easily seen portions of theID tags 130, particularly if the object of interest is oriented in amanner relative to the AR headset 115 that is difficult for thecamera(s) 180 of the AR headset 115 to capture images or video that iseasily identifiable by the system. In some embodiments, the frontcamera(s) 180 of two or more AR headsets 115 may be used to captureimages or videos of different areas or perspectives of the workenvironment 135 to achieve a similar functionality as the use ofcamera(s) 195 in conjunction with front camera(s) 180 of one of the ARheadsets 115. In some cases, the images or video captured by thecamera(s) 180 of the two or more AR headsets 115 together with theimages or video captured by the camera(s) 195 may be analyzed by theprocessor 150, the computing system 105 a or 105 b, and/or the AI system140 to identify objects of interest among the objects 125, therebyfurther facilitating identification of objects of interest due to theincreased number of views or perspectives afforded by the camera(s) ofmultiple AR headsets and the area camera(s) 195.

In operation, at least one of the front camera(s) 180 of a first ARheadset 115 a, the front camera(s) 180 of one or more other AR headsets115 b-115 n, and/or the camera(s) 195 (collectively, “image capturedevice(s)” or “camera(s)” or the like) might capture images or video ofat least a first portion of work environment 135. In some cases,(particularly, with the images or videos captured by the front camera(s)180 of the first AR headset 115 a) the captured images or video mightoverlap with a field of view of the eyes of the first user 120 a wearingthe first AR headset 115 a. The image capture device(s) mightsubsequently send the captured images or video. At least one of theprocessor 150 of the first AR headset 115 a, the computing system 105 a,the remote computing system 105 b, and/or the AI system 140(collectively, “the computing system” or the like) might receive thecaptured images or video (collectively, “captured images” or the like)from the image capture device(s), might analyze the received capturedimages to identify one or more first objects 125 among the one or moreobjects 125 a-125 n, might query at least one database (e.g., data store155, database(s) 110 a, and/or database(s) 110 b, or the like) todetermine a first task associated with at least one first object amongthe identified one or more first objects 125, and might generate a firstimage overlay. In some embodiments, the first image overlay mightprovide at least one of a graphical icon-based instruction, a text-basedinstruction, an image-based instruction, or a highlighting-basedinstruction, and/or the like, each indicative of one or moreinstructions to be presented to first user 120 a to implement the firsttask associated with the at least one first object 125. The computingsystem might then display, to the eyes of the first user 120 a throughthe AR headset, the generated first image overlay that overlaps with thefield of view of the eyes of the first user 120 a.

In some embodiments, particularly in cases where the speaker(s) orearpiece(s) 160 is part of the AR headset 115 a, the computing systemmight generate one or more audio-based instructions (which mightinclude, but is not limited to, at least one of a tone, a series oftones, spoken instructions, or test-to-speech content, and/or the like)corresponding to each of at least one of the one or more instructions tobe presented to the user via the generated first image overlay, andmight present (via the speaker(s) or earpiece(s) 160 of AR headset 115a) the generated one or more audio-based instructions in conjunctionwith displaying corresponding each of at least one of the one or moreinstructions being presented to the user via the generated first imageoverlay as displayed to the eyes of the first user 120 a through the ARheadset 115 a.

In accordance with the various embodiments described herein, theaugmented reality (“AR”) system provides for generation of imageoverlays (including, but not limited to, at least one of graphicalicon-based overlays, text-based overlays, image-based overlays, orhighlighting-based overlays, and/or the like) that are superimposed overeither an actual view of an area (e.g., work environment or the like) oran image or video of the area. In the former case, the actual view ofthe area might be viewed by the user through a lens, visor, transparentscreen, or the like, with the generated image overlays either beingprojected directly into the eyes of the user, being projected on aprojection plane or surface in front of the eyes of the user (e.g., onthe plane or surface defined by the lens, visor, transparent screen, orthe like), or being displayed on the surface of the lens, visor, ortransparent screen, or the like. Alternatively, for AR headsets withouta lens, visor, transparent screen, or other component disposed betweenthe eyes of the user (or the user's personal corrective lenses), theactual view of the area might be viewed directly by the eyes of the user(or through only the user's personal corrective lenses), with thegenerated image overlays either being projected directly into the eyesof the user or being projected as a hologram in front of the eyes of theuser, or the like. In the latter case, the image or video of the areamight be viewed by the user using a display screen disposed in front ofthe user (which might include, but is not limited to, a display screenintegrated within the AR headset or a display screen of an AR-enabledsmart phone that is mounted in the AR headset in front of the eyes ofthe user, etc.), with the generated image overlays being superimposedover the image or video of the area prior to the composite image orvideo being displayed on the display screen. In the various embodiments,the generated image overlays are superimposed over the image or video ofthe area, with the image or video being displayed in real-time ornear-real-time. The use of the AI system 140 or other computing processenhancers or parallel processors to reduce the computational time foridentification of objects of interest, querying databases to determinetasks associated with the objects of interest, generation of imageoverlays based on the determined tasks in relation with the objects ofinterest, and display or projection of the generated image overlays tothe user, in order to achieve the real-time or near-real-time ARexperience for the user.

According to some embodiments, the at least one of the graphicalicon-based instruction, the text-based instruction, the image-basedinstruction, or the highlighting-based instruction, and/or the like,might include, but is not limited to, at least one of a graphicalicon-based instruction, a text-based instruction, an image-basedinstruction, or a highlighting-based instruction that, when displayedwithin the generated first image overlay, is superimposed over,displayed around, or displayed beside the at least one first object asviewed by the user through the AR headset. Merely by way of example, thegraphical icon-based instruction might include, without limitation, atleast one of a graphical icon representing identification informationassociated with the at least one first object; a graphical iconrepresenting identification information associated with a second objectwith which the at least one first object is intended to interact as partof the first task; a colored graphical icon distinguishing one firstobject from another first object among the at least one first object; agraphical icon comprising a directional arrow representing a directionthat the at least one first object should be taken as part of the firsttask; or a colored graphical icon comprising a colored directional arrowdistinguishing a first direction that the at least one first objectshould be taken as part of the first task from a second direction that athird object should be taken as part of the first task, and/or the like.

In some instances, the text-based instruction might include, but is notlimited to, at least one of a floating text window comprising textualinstructions corresponding to the first task; or a surface text windowthat is superimposed on one of a table-top surface, a wall surface, oran object surface and that comprises textual instructions correspondingto the first task. In some cases, the floating text window, whendisplayed within the generated first image overlay, is displayed as afloating image beside the at least one first object or displayed as afloating image within the field of view of the eyes of the first user.In some instances, the image-based instruction might include, withoutlimitation, at least one of an image representing identificationinformation associated with the at least one first object; an imagerepresenting identification information associated with a second objectwith which the at least one first object is intended to interact as partof the first task; an image comprising a directional arrow representinga direction that the at least one first object should be taken as partof the first task; an image comprising images of numbers or codesrepresenting an order of processes of the first task associated with theat least one first object; a magnified image of the at least one firstobject; a three-dimensional (“3D”) image or hologram; or an image of theat least one first object superimposed over a targeted portion of thework environment indicating at least one of position, orientation, orconfiguration of the at least one first object intended for placement atthe targeted portion; and/or the like. In some cases, thehighlighting-based instruction might include, but is not limited to, atleast one of a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over the at least onefirst object; a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a fourth object thatis related to the first task associated with the at least one firstobject; or a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a targeted portionof the work environment indicated intended placement of the at least onefirst object at the targeted portion; and/or the like.

In some cases, the light source(s) 170 of the first AR headset 115 amight illuminate one or more of the at least one first object 125 a orthe at least a first portion of the work environment 135 during at leastone of the following periods: prior to the one or more first imagesbeing captured by the image capture device(s); or in conjunction withthe display of the generated first image overlay; and/or the like.

In some embodiments, the computing system, using the eye trackingsensor(s) 165 that are disposed on the first AR headset 115 a and thathave a field of view that includes the eyes of the first user 120 a(i.e., that are directed at the eyes of the first user 120 a, or thelike), might track the eyes of the first user 120 a to determine adirection in which the first user is looking, and might determine acentral portion of a field of view of the first user (i.e., determinewhat the first user 120 a is looking at, or the like) based at least inpart on the determined direction that the first user 120 a is looking inconjunction with the captured images as captured by the image capturedevice(s). In some instances, in response to determining the centralportion of the field of view of the first user (i.e., in response todetermining what the first user is looking at, or the like), thecomputing system might identify at least one fifth object (in a mannersimilar to the process described above with regard to identification ofthe one or more first objects).

In various aspects, the first task might comprise sorting, categorizing,arranging, or organizing the one or more first objects, wherein the oneor more objects might comprise one or more microscope slides eachcontaining a test sample or a tissue sample (among other laboratoryinstruments, tools, objects, or equipment), as depicted in thenon-limiting examples of FIGS. 2A and 2B. Alternatively, the first taskmight include, without limitation, one of running a test sample or atissue sample using laboratory instrumentation, solving a puzzle,assembling a piece of machinery, assembling an object, preparing a testsample or a tissue sample, identifying and/or tracking samples within alaboratory, instructing or guiding users regarding how to performexperimental protocols or tests in a laboratory, performing processes ina histology laboratory, performing processes in a pathologist's office,performing tasks after slide diagnosis, performing processes in aclinical or analytical laboratory, or transferring reagents or samples,and/or the like. In such cases, the one or more first objects mightinclude, but are not limited to, one of laboratory instrumentation,laboratory tools, sample transfer devices, puzzle components or pieces,machinery parts, assembly tools, measurement tools, object parts, samplereagents, sample containers, burners, coolers, mixers, samplepreparation tools, or sample transfer tools, and/or the like. In somecases, a plurality of users 120 a-120 n may be tasked with performingthe first task (as depicted in FIG. 3A, or the like), where each user120 wears an AR headset 115 that displays generated image overlays toeach user 115 that distinguishes objects that one user is intended tointeract with from objects that the other users among the plurality ofusers are intended to interact with as part of the first task (asdepicted in FIG. 3B, or the like). In some embodiments, the workenvironment might include, without limitation, one of a laboratory, aconstruction site, a machine shop, a workshop, a factory, or a room,and/or the like.

These and other functions of the system 100 (and its components) aredescribed in greater detail below with respect to FIGS. 2-4.

FIGS. 2A and 2B (collectively, “FIG. 2”) are schematic diagramsillustrating a set of non-limiting examples 200 and 200′ of sorting,categorizing, arranging, or organizing of objects performed by a userutilizing an AR headset that is used for implementing AR-basedassistance within a work environment, in accordance with variousembodiments. FIG. 2A depicts a non-limiting example 200 of a user'sperspective looking through an AR headset that either allows the user tosee the actual work environment or displays images or video of theactual work environment, without use of AR image overlays, while FIG. 2Bdepicts a non-limiting example 200′ of the first user's perspectivelooking through the AR headset that includes a generated AR imageoverlay superimposed over either the actual work environment or thedisplayed images or video of the actual work environment. Sorting ofslides is described in greater detail in the '122 Application (which hasalready been incorporated herein by reference in its entirety for allpurposes), in which “artificial reality” refers to “augmented reality”as described herein. Although the sorting, categorizing, arranging, ororganizing of objects is shown with respect to FIG. 2 as being a taskthat may be enhanced by the use of AR-based assistance or functionalityas described herein, the various embodiments are not so limited, and thetask that may be enhanced by the use of AR-based assistance orfunctionality may alternatively include, without limitation, one ofrunning a test sample or a tissue sample using laboratoryinstrumentation, solving a puzzle, assembling a piece of machinery,assembling an object, preparing a test sample or a tissue sample,identifying and/or tracking samples within a laboratory, instructing orguiding users regarding how to perform experimental protocols or testsin a laboratory, performing processes in a histology laboratory,performing processes in a pathologist's office, performing tasks afterslide diagnosis, performing processes in a clinical or analyticallaboratory, or transferring reagents or samples, and/or the like (notshown), to be performed in corresponding one of a laboratory, aconstruction site, a machine shop, a workshop, a factory, or a room,and/or the like, where objects of interest within such work environmentsmight include, but are not limited to, one of laboratoryinstrumentation, laboratory tools, sample transfer devices, puzzlecomponents or pieces, machinery parts, assembly tools, measurementtools, object parts, sample reagents, sample containers, burners,coolers, mixers, sample preparation tools, or sample transfer tools,and/or the like.

According to some embodiments, examples 200 and 200′ of FIGS. 2A and 2Bmight comprise an AR headset 205, one or more front or front-facingcameras 210 a or 210 b (collectively, “cameras 210” or “image capturingdevices 210” or the like), one or more eye tracking sensors 215 a or 215b (collectively, “eye tracking sensors 215” or “eye tracking cameras215” or the like), and/or the like.

Prior to AR functionality being implemented, a view of a portion of thework environment (e.g., a laboratory, a construction site, a machineshop, a workshop, a factory, or a room, or the like) may be seen by auser when wearing the AR headset 205. In some cases, such a view may beprovided through a viewport (e.g., a lens, visor, or transparent screen,or the like) of the AR headset 205, or via a display screen (e.g., adisplay screen integrated within the AR headset 205, or a display screenof an AR-enabled smart phone that is mounted in the AR headset 205 infront of the eyes of the user, or the like) displaying an image or videoof the portion of the work environment. For instance, referring to FIG.2A, a view of a portion of a work environment (in this case, alaboratory, or the like) is seen by a user through the AR headset 205.Within such a view might be a table or counter 220, on which might bedisposed a plurality of microscope slides 225 to be sorted, categorized,arranged, or organized as part of a task to be completed by a user,where each slide 225 might have affixed thereto an identifier oridentification (“ID”) tag 225 a. In the non-limiting example of FIG. 2,each microscope slide 225 might contain a sample 230 for testing,cataloguing, or other laboratory or scientific purpose. As part of thetask associated with the slides 225, a user may be tasked with sorting,categorizing, arranging, or organizing the slides 225, which mightinvolve placing particular slides (or sets/groups of slides) inparticular trays 235 or containers 240 (each type of which, in somecases, may be stackable, as shown in FIG. 2), each tray 235 or container240 having an ID tag 235 a or 240 a, respectively, affixed thereto. Insome cases, the table or counter 220 might also have an ID tag 220 aaffixed thereto.

In some instances, the ID tags 220 a, 225 a, 235 a, and/or 240 a mighteach comprise a unique identifier, which might include, withoutlimitation, a unique numerical identifier, a unique alphanumericidentifier, a unique symbolic identifier, a unique one-dimensional(“1D”) visual code (including, but not limited to, a unique bar code, orother unique 1D pattern code, or the like), a unique two-dimensional(“2D”) visual code (including, but not limited to, a unique quickresponse (“QR”) code, or other unique 2D pattern or matrix code, or thelike), a unique three-dimensional (“3D”) visual code (including, but notlimited to, a unique 3D pattern code, a unique 3D matrix code, a unique3D holographic code, a unique 3D holographic pattern code, a unique 3Dholographic matrix code, or the like), a digital image of at least aportion of the object itself or of a representative example of theobject (e.g., digital image of at least a portion of a tissue sample ortissue section, table, tray, container, instrumentation, tool, or otherobject, etc.), or a unique identifying feature or attribute (of anobject among one or more objects 220, 225, 235, 240, etc.) (shown inFIG. 2), and/or the like. Alternatively, or additionally, at least oneof the ID tags 220 a, 225 a, 235 a, and/or 240 a might each comprise aradio frequency identifier (“RFID”) tag, or the like, that wheninterrogated by a radio frequency scanner (which may be disposed in theAR headset 205 (not shown in FIG. 2), or may be disposed within adedicated RFID scanner or other similar device (not shown in FIG. 2))would broadcast identification information (which might include theunique identifier data, or the like) in rf response signals. [Thisdescribes a passive RFID tag, although a battery-assisted passive RFIDtag may be used. An active RFID tag that broadcasts its data withoutneed of power from an interrogating signal from a rf scanner (in somecases, broadcasting constantly) could be used, but within the slidesorting, categorizing, arranging, or organization context, the computingsystem (such as described above with respect to FIG. 1) may be requiredto distinguish amongst the multitude of rf signals from the plurality ofslides. A rf scanner having a narrow field of rf interrogation forinterrogating passive or battery-assisted passive RFID tags within alimited area would avoid the problem inherent with active RFID tags forsuch a task. For other tasks that do not involve many RFID tags within alimited area, active RFID tags may function as well or better thanpassive or battery-assisted passive RFID tags. Alternatively, even fortasks that involve many RFID tags within a limited area (such as theslide sorting, categorizing, arranging, or organization of objects,etc.), the use of gaze focusing techniques as described below (i.e.,tracking what the central portion of the field of view of the user todetermine what the user is looking at) or the use of gestures by theuser's hand(s) (i.e., having the user point directly at a particularobject(s) having an RFID, or the like) or the use of gestures by apointer or selection device(s) operated by the user, and/or the like,may serve as a trigger to filter out (in any generated image overlays)active RFID tag information associated with objects that are outside ofthese areas (i.e., outside of the central portion of the field of viewof the user, outside the area where the user is pointing with the user'shand(s), or outside the area where the user is pointing using thepointer or selection device, etc.).] In some embodiments, a smart tagthat combines RFID tag functionality with printed ID functionality(e.g., with use of barcodes or other 1D visual codes, QR codes or other2D visual codes, 3D visual codes, numbers, letters, text, code, etc.)may also be used. Alternative, or additional, to the use of ID tags 220a, 225 a, 235 a, and/or 240 a for identification of objects (such as thetables or counters 220, slides 225, the trays 235, or the containers240, and/or the like), image recognition or photogrammetric recognitionfunctionality (particularly as enhanced by use of an AI system or thelike) may be used to perform, or aid in the performance of,identification of objects of interest among the objects.

With reference to FIG. 2B, AR-based assistance may be implemented (anon-limiting example 200′ of which is shown in FIG. 2B). In particular,the cameras 210 a and/or 210 b might capture an image(s) or video(collectively, “captured images” or the like) of the portion of the workenvironment (e.g., laboratory in this case, or the like). A computingsystem (such as at least one of the processor 150 of the first ARheadset 115 a, the computing system 105 a, the remote computing system105 b, and/or the AI system 140 of FIG. 1, or the like, as describedabove) might receive the captured images from the cameras 210 a and/or210 b, might analyze the captured images to identify one or more firstobjects among the objects present in the captured images of the workenvironment, might query a database (e.g., data store 155, database(s)110 a, and/or database(s) 110 b of FIG. 1, or the like) to determine afirst task associated with at least one first object among theidentified one or more first objects, and might generate a first imageoverlay. In some embodiments, the first image overlay might provide atleast one of a graphical icon-based instruction, a text-basedinstruction, an image-based instruction, or a highlighting-basedinstruction, and/or the like, each indicative of one or moreinstructions to be presented to the user to implement the first taskassociated with the at least one first object. The computing systemmight then display, to the eyes of the user through the AR headset 205,the generated first image overlay that overlaps with the field of viewof the eyes of the first user. Turning to FIG. 2B, the first imageoverlay might include, but is not limited to, a first series ofinstructions 245, a second series of instructions 250, and a thirdseries of instructions 255, and/or the like.

In some embodiments, such as depicted in FIG. 2B, the first series ofinstructions 245 might include, without limitation, a graphical icon orimage in the shape of a microscope slide outline 245 a that, whendisplayed in the first image overlay, appears to the user to surround afirst slide 225 among the plurality of slides 225; an image of an arrow245 b that points from the first slide 225 to a particular slot within afirst container 240 (denoted as “Folder 1” in FIG. 2B); and a hologramor image of a slide 245 c fitting within the particular slot withinfirst container 240; or the like. In a similar manner, the second seriesof instructions 250 might include, but is not limited to, ahighlighting-based field 250 a that covers a second slide 225 among theplurality of slides 225; an image of an arrow 250 b that points from thesecond slide 225 to a particular indentation, cut-out, or bed within atray 235 (denoted as “Tray 1” in FIG. 2B); and a highlighting-basedfield 250 c that covers the particular indentation, cut-out, or bedwithin the particular tray 235; or the like. Likewise, the third seriesof instructions 255 might include, without limitation, a graphical iconor image in the shape of a microscope slide outline 255 a that, whendisplayed in the first image overlay, appears to the user to surround athird slide 225 (denoted as “Slide 109” in FIG. 2B) among the pluralityof slides 225; an image of an arrow 255 b that points from the thirdslide 225 to a particular slot within a third container 240 (denoted as“Folder 3” in FIG. 2B); and a highlighting-based field or image 255 coutlining the bottom portion of the particular slot within thirdcontainer 240; or the like.

Although the first through third sets of instructions are depictedtogether in FIG. 2B, this is done simply for illustration of some (butnot all) of the different types of instructions that may be displayed ina graphical manner to the user to indicate one or more tasks to becompleted by the user. The various embodiments are not so limited,however, as these types of instructions may be provided serially (i.e.,one after the other, perhaps in priority according to some presetcriteria). In some cases, the concurrent or serial sets of instructionsmight have consistent characteristics or features in accordance witheither default display settings or preferences of the user, which mightbe conditioned based on one or more of the following: the types ofobjects, ease of illustration, the need to avoid confusion, and/or thelike. In some instances, some (or all) of the icons, images,highlighting fields, text fields, and/or the like might be made toflash, blink, vibrate, and/or change color to attract the attention ofthe user, or to emphasize importance or priority, or the like. In somecases, some (or all) of the icons, images, highlighting fields, textfields, and/or the like might be made to appear to be projected on asurface (e.g., table-top surface, wall surface, surface of an object(which can be but need not be flat), or projected as a floating hologramor floating window (or info screens), or the like, that are eitherstationary or movable to track motion of objects, or the like. Althoughnot shown, icons including, but not limited to, biohazard icons,flammable caution icons, caustic caution icons, poisonous caution icons,and/or the like may be generated in the image overlays to hover above,around, beside, or near particular objects exhibiting the correspondingtrait, or the like.

In some embodiments, rather than instructions, information may bedisplayed; for purposes of simplicity of description, however, suchinformation is also referred to herein as instructions. For example,text-based tags 260 may be displayed, as an image overlay, beside oraround some or all of relevant objects within the work environment. Thetext-based tags 260 might include, for instance, a slide-identifyingtext-based tag 260 a (e.g., identifying “Slide 109” or the like), acontainer-identifying text-based tag 260 b (e.g., identifying “Folder 1”or the like), another container-identifying text-based tag 260 c (e.g.,identifying “Folder 3” or the like), and a tray-identifying text-basedtag 260 d (e.g., identifying “Tray 1” or the like). According to someembodiments, the computing system might enhance, modify, or otherwisechange an image of the object of interest. For instance, an image of thethird slide 225 (i.e., “Slide 109”) might be image-processed to enlarge(or magnify) the original image (to effect a “zoomed-in” feature) and torotate, pan, and/or tilt the image to present a top view of the thirdslide 225 displayed vertically, in order to provide the user with aclose-up view of the third slide (or object of interest). Suchimage-processing may be initiated in response to selection by the user(e.g., by gazing intently at the third slide, by gazing at the thirdslide coupled with verbal instructions to “focus-in” on the third slide(or the like), by selecting using a virtual cursor (or the like; notshown), by pointing at the third slide with the user's finger (or thelike; not shown), etc.). Alternative, or additional, to the display ofclose-up view of the third slide (or object of interest), the user mightselect to display additional information about the third slide, suchinformation either being decoded from the ID tag 225 a of the thirdslide 225 or retrieved from the database based on the unique identifierinformation embedded within the ID tag 225 a. In some cases, suchinformation might include sample number (e.g., “Sample #109” or thelike) and a name of a person working with the sample on the slide (e.g.,“Dr. T. Who” or the like), which may be displayed as a text-based imageoverlay 270 a. In some instances, the user might seek more detailedinformation about the object—in this case, sample unique ID (e.g.,“CK120118-109”), sample type or name (e.g., “Cytokeratin CE1/CE3” or thelike), and name of a person working with the sample on the slide (e.g.,“Dr. T. Who” or the like), which may be displayed as a text-based imageoverlay 270 b. Although particular examples of image overlays areillustrated in FIG. 2B, the various embodiments are not so limited, andany suitable type, shape, configuration, orientation, color, or size ofimage overlays may be used consistent with the embodiments describedherein.

Also, although not shown, the image overlay may include instructions forthe user to move a tray or container from one location to another(before or after being filled with slides), to move from one location(e.g., table or counter) to another, to change the order, orientation,or arrangement of trays or containers, and/or the like. Further, thecomputing system might continuously, periodically, or upon a conditionbeing met (e.g., filling of at least 50% or 75% of a tray or container,or the like) identify the slides placed in the particular tray orcontainer, and determine whether the correct slides are placed in thecorrect tray or container (and in the correct cut-outs or slots). If amistake has been made, the computing system might generate and displayan image overlay with instructions to the user to make the correction.Moreover, in addition to the visual aspect of the system, as illustratedby the AR image overlays, the computing system might interact with theuser via aural or verbal communications (e.g., using the speaker(s) orearpiece(s) 160 and the audio sensor(s) 175 of FIG. 1, or the like, asdescribed above). For instance, upon completion of a task, or sub-task,the computing system might generate and display image overlaysindicating successful completion of the task or sub-task (in some cases,in conjunction with verbal communication indicating the same).

According to some embodiments, the eye tracking sensors 215 may be used(perhaps in conjunction with audio commands as monitored by audiosensors (such as audio sensor(s) 175 of FIG. 1, or the like; not shownin FIG. 2)) to register or track commands issued by the user. Forexample, the user might gaze at a certain folder for some period of timeexceeding a predefined amount of time (e.g., 3 seconds, or the like),which might indicate that the user desires to select this particularfolder, resulting in the selected folder being highlighted by an imageoverlay (e.g., a highlighting-based field, floating hologram, or otherappropriate image, or the like). Alternatively, the user might look at aparticular folder and might concurrently say the command, “Select,”which might indicate that the user desires to select this particularfolder, resulting in the selected folder being highlighted by an imageoverlay (e.g., a highlighting-based field, floating hologram, or otherappropriate image, or the like) as above. In another alternative, theuser might look at a particular folder and might perform a gesture withthe user's hand(s) or a pointer or selection device(s) (in accordancewith predefined characteristics or parameters for a selection gesture),which indicate that the user desires to select this particular folder,resulting in the selected folder being highlighted by an image overlay(e.g., a highlighting-based field, floating hologram, or otherappropriate image, or the like) as above. Once a folder is selected, theuser can ask for additional information (e.g., “how many slides aremissing to close this folder?” or “who is the doctor assigned to thisparticular slide?”, etc.). In another example interaction, an incomingrequest from Dr. Who might indicate to finish the folder for Patient XASAP, in which case, a technician or user might say, “show me Patient Xfolder,” or might select the folder using one of the methods indicatedabove. Once the technician or user has selected the folder, he or shecan say, “High priority” or similar phrase, which would result in thecomputing system elevating the priority of the folder, and generatingand displaying image overlays (perhaps in conjunction with verbalinstructions) to fill the folder first with appropriate marking orhighlighting of the applicable or relevant slides to finish the folderfor Patient X.

In the case that the computing system cannot properly identify anobject, the computing system might generate and display an image overlay(e.g., a highlighting field) displayed so as to appear to the user tocover or highlight the object, with text and/or verbal instructions tothe user to take a closer look at the marked slide (e.g., by movingcloser, by orienting the user's head for a better angle, by removing anyobstructing object, by picking up the marked object, and/or the like).In some cases, the user may be instructed in this manner to read out anyidentifiers (e.g., numeric codes, alphanumeric codes, text, and/or thelike).

The various features or characteristics of the AR-based assistanceimplementation (and the various image overlays and instructions)described above with respect to FIG. 2, although directed to the task ofsorting, categorizing, arranging, or organizing of objects (inparticular, slides 225), may similarly be applicable to sorting,categorizing, arranging, or organizing of other objects (either in alaboratory or in other settings or work environments) or may likewise beapplicable to other tasks, including, but not limited to, running a testsample or a tissue sample using laboratory instrumentation, solving apuzzle, assembling a piece of machinery, assembling an object, preparinga test sample or a tissue sample, identifying and/or tracking sampleswithin a laboratory, instructing or guiding users regarding how toperform experimental protocols or tests in a laboratory, performingprocesses in a histology laboratory, performing processes in apathologist's office, performing tasks after slide diagnosis, performingprocesses in a clinical or analytical laboratory, or transferringreagents or samples, and/or the like, as applicable, as appropriate, oras desired.

For instance, in a non-limiting example of the use case involvingrunning the sample using laboratory instrumentation, the computingsystem might identify (by the techniques described herein) which modelof instrumentation (e.g., which particular model of a mass spectrometer,a gas chromatography system, a liquid chromatography system, etc.) andin response to the user selecting or saying what operation he, she, orthey would like to run (e.g., “Run mass spectrometry analysis on SampleT”; etc.), the computing system might access a user manual of theparticular model of instrumentation, and might generate and displayimage overlays (perhaps also generate and present aural instructions) tothe user indicating how to operate the particular piece ofinstrumentation to complete the task as indicated, and also indicate(using image overlays and perhaps also aural instructions regarding)when, where, and how to prepare the sample for performing the indicatedanalysis and to indicate (using image overlays and perhaps also auralinstructions regarding) when, where, and how to insert the sample intothe piece of instrumentation to run the analysis, and to also indicate(using image overlays and perhaps also aural instructions regarding)what buttons, switches, valves, etc. to actuate and when to do so.

In cases where there is an area camera whose field of view overlaps withthe display screen of the instrumentation, the computing system mightalso generate and display an image overlay that is either a direct imagecopy or a floating window with information displayed on theinstrumentation display screen being scraped and filled into appropriateportions of the floating window. In this manner, the user can move toother areas of the laboratory (i.e., work environment) to perform othertasks or sub-tasks (which may or may not be related to the indicatedtask (e.g., preparing the sample for performing the task while theinstrumentation is warming up, etc.), while being aware from theinformation displayed in the floating window image overlay (which may bemoved to a side or corner of the display area within the AR headset,based on commands by the user (e.g., gaze selection and gaze movement,gaze and verbal phrase selection and movement, selection using afloating mouse or other device, etc.)). Similarly, image overlays may begenerated and displayed to provide information to the user, including,but not limited to, when the process by the instrumentation has beensuccessfully completed, when errors occur together with tips andguidance (in the form of text or holograms, etc.) to address the errors,or information regarding the samples as the task is being performed(e.g., real-time temperature, average temperature, expected color changeunder particular reactions and whether the sample's color change iswithin expected color spectrum range, expected spectrographs orchromatographs for the particular sample and whether the sample's actualresultant spectrograph or chromatograph is within expected ranges,etc.), and/or the like. In some cases, the computing system mightmonitor use of instrumentation in the laboratory by other users andmight indicate expected wait times for the user to know when suchinstrumentation is available. These are merely some examples of AR-basedassistance that can improve efficiency and optimization in a laboratory,e.g., when using instrumentation to run test or tissue samples, and donot limit the various embodiments to such examples or for such uses ofinstrumentation.

In the case of solving a puzzle, a non-limiting example might be solvinga jigsaw puzzle or the like. In such an example, when the user hasjoined some pieces of the jigsaw puzzle together, but is stuck in termsof other pieces to join to the partially assembled puzzle, the usermight say, “Help” or “Hint,” which might trigger the computing system toidentify (by the techniques described herein) a random piece among aplurality of pieces that can be joined to the one or more pieces thathave already been joined to form the partially assembled puzzle, and togenerate and display image overlays that highlight the identified randompiece, that highlight the spot that the identified random piece wouldfit with the one or more pieces of the partially assembled puzzle, andthat points (with a holographic arrow or the like) from the highlightedrandom piece to the highlighted spot. Although assistance with solving ajigsaw puzzle is described, the various embodiments are not so limited,and the AR assistance may be implemented to aid in solving any othertype of puzzle.

In the case of assembling a piece of machinery or assembling an object,the computing system might identify (by the techniques described herein)the parts of the machinery or object, might query a database forassembly instructions, and might generate and display image overlays(and perhaps also generate and present aural instructions) that outlinethe steps (e.g., in a floating window with appropriate still diagrammontages or dynamic diagrams showing how one piece is fitted withanother, etc.) while also highlighting parts and tools for each step,until the machinery or object has been successfully assembled, at whichpoint the computing system might generate and display image overlays(and perhaps also generate and present aural notifications) indicatingsuccessful assembly of the machinery or object. In situations where oneor more parts are defective, broken, or simply the wrong part, thecomputing system might generate and display image overlays (and perhapsalso generate and present aural notifications) indicating such and, insome cases, also presenting the user with options to order new parts orto request replacement and/or refund of the machinery or object.Although a particular set of examples is described with respect to ARassistance with assembly of machinery or object, the various embodimentsare not so limited, and the AR assistance may be implemented in otherways to assist in the assembly of machinery or objects, consistent withor in accordance with the various embodiments described herein.

In the case of preparing a test sample or tissue sample, a non-limitingexample might involve the computing system identifying (by thetechniques described herein) which sample, which microscope slides,which labelling machine, etc. to use, and might generate and displayimage overlays (and perhaps also generate and present auralinstructions) indicating use of a micropipette to transfer theidentified sample from a vial or tube to a microscope slide, indicatinghow to stain the sample, indicating how to place another microscopeslide to sandwich the stained sample, and indicating how to label thecombined slide with an ID tag (such as ID tag 225 as shown in FIG. 2, orthe like), in some cases, with the use of highlighting fields,holographic arrows, still diagram montages, and/or moving or dynamicdiagrams depicting each step of the preparation. Although a particularset of examples is described with respect to preparation of a sample inmicroscope slides, the various embodiments are not so limited, and theAR assistance may be implemented in other ways to prepare test or tissuesamples, such as preparation of samples that do not involve the use ofmicroscope slides, including, but not limited to, preparing samples formass spectrometry analysis, preparing samples for gas chromatographyanalysis, preparing samples for liquid chromatography analysis, etc.,consistent with or in accordance with the various embodiments describedherein.

In the case of identifying and/or tracking samples within a laboratory,a non-limiting example might involve the computing system uniquelyidentifying (by the techniques described herein) a sample when itarrives at a laboratory and is subsequently logged in. In response to acustomer request being logged in, a laboratory director, a laboratorytechnician, or an automated standard protocol, or the like might assignappropriate laboratory processes needed to analyze the sample. Thecomputing system might generate and display image overlays (and perhapsalso generate and present aural instructions) to users or technicianswearing the AR headsets to identify the sample and to instruct the usersor technicians on the current and next appropriate steps in the assignedprocesses, regardless of whether the sample changes form during any partof the analytical or preparative processes. In some cases, the imageoverlays might include use of highlighting fields, holographic arrows,still diagram montages, and/or moving or dynamic diagrams depicting eachstep of the assigned processes. In some instances, the users ortechnicians might use voice input or the like (e.g., by using amicrophone or other audio sensors, such as audio sensor(s) 175 of FIG.1, or the like) to perform one or more of: changing or correcting thenames of the samples; entering, changing, or updating other informationabout the samples (e.g., composition information, storage information,physician information, laboratory information, etc.); changing one ormore steps in the assigned processes; suggesting or changing processesassociated with downstream processes involving the samples; etc.Although a particular set of examples is described with respect toidentifying and/or tracking samples within a laboratory, the variousembodiments are not so limited, and the AR assistance may be implementedin to track other objects other than samples and to do so in workenvironments other than a laboratory, consistent with or in accordancewith the various embodiments described herein.

In the case of instructing or guiding users regarding how to performexperimental protocols or tests in a laboratory, a non-limiting examplemight involve the computing system identifying (by the techniquesdescribed herein) objects and instrumentation that is present in alaboratory. The computing system might generate and display imageoverlays (and perhaps also generate and present aural instructions)indicating what one or more objects or instrumentation in the laboratoryare and what they might be used for, indicating how the one or moreobjects might be tested or used in laboratory tests or processes,indicating how the instrumentation is used to test samples, to preparesamples, or to perform other laboratory processes, and indicating othersteps in the experimental protocols or tests, in some cases, with theuse of highlighting fields, holographic arrows, still diagram montages,and/or moving or dynamic diagrams depicting each object,instrumentation, step, or process. Although a particular set of examplesis described with respect to instructing or guiding users regarding howto perform experimental protocols or tests in a laboratory, the variousembodiments are not so limited, and the AR assistance may be implementedto instruct or guide users regarding how to perform other procedures ina laboratory or to perform other tasks that are not based in alaboratory, consistent with or in accordance with the variousembodiments described herein.

In the case of performing processes in a histology laboratory, anon-limiting example might involve the computing system identifying (bythe techniques described herein) a tissue sample that arrives at apathology laboratory. In the pathology laboratory, the tissue samplemight be assigned a unique identifier and tests defined by the clinicianor pathologist might be logged. The tissue sample and an appropriatecassette(s) that is(are) used to contain the tissue sample might bemoved to grossing where a pathologist or technician wearing an ARheadset might examine and document the tissue sample, might dictate (byusing a microphone or other audio sensors, such as audio sensor(s) 175of FIG. 1, or the like) his, her, or their observations, might selectand size the tissue sample for submission to histology processes, andmight place the tissue sample(s) in the identified cassette(s), allalong aided, assisted, and/or guided by image overlays that aregenerated and displayed through the AR headset (which might alsogenerate and present aural instructions). Next, a technician wearing anAR headset might be guided (via generated and displayed image overlaysand/or generated and presented aural instructions through the ARheadset) to move the identified cassette(s) to a processing stationwhere the tissue sample might be fixed, dehydrated, and infiltrated inparaffin per the appropriate protocol for the type and size of thattissue sample. After processing, the technician might embed the tissuesample in paraffin to form a tissue block on a backside of theidentified cassette(s). The computing system might generate and displayimage overlays (and perhaps also generate and present auralinstructions) to guide the technician with any tissue sample-specificinstructions determined from the grossing step or from standardprotocol. The computing system might accept any sample-specificinstructions that are needed for subsequent steps, such as the number oftissue slides required as dictated (via the microphone or other audiosensors, etc.) by the technician performing the embedding.

The computing system might document (via the front camera(s) 180 or 210a or 210 b of FIG. 1 or 2, or the like) the face of the tissue block forsubsequent quality control steps. Next, the technician might section thetissue block and might place the sections on appropriately identifiedmicroscope slides (or other sample containers). The technician may thenadd information to the system regarding how many slides were created andif any difficulties or issues were observed (such as insufficient tissueor the sample needed to be returned for additional processing). Thecomputing system can compare the unique identifier for the block, theunique identifier for the microscope slide, the block face (or imagethereof), and/or the tissue section(s) (or image(s) thereof) forconsistency. Next, the technician might stain the slides per the correctassay, either by being guided (via the AR headset) through the chemicalprotocol, or by bringing the slides to the appropriate automated systemthat has the reagents and the capacity to proceed with the assay. Afterstaining, the technician either might place the slides in the correctfolder for delivery to a pathologist via the sorting process describedabove with respect to FIGS. 2A and 2B or might scan the slides forelectronic delivery to the pathologist. The computing system mightautomatically compare the unique identifier for the slide and the imageof the stained section to the original tissue block for consistency. Thecomputing system can also be used to infer a sample location. Forexample, microscope slides are often grouped in baskets for automatedprocessing. If a specific slide is placed in the basket with others,when one slide is identified in the basket at another location, thelocation of all other slides in that basket can be inferred even if theunique identifier for those other slides cannot be visually confirmed.Although a particular set of examples is described with respect toperforming processes in a histology laboratory, the various embodimentsare not so limited, and the AR assistance may be implemented in otherways to perform other tasks in a histology laboratory, consistent withor in accordance with the various embodiments described herein.

In the case of performing processes in a pathologist's office, anon-limiting example might involve the computing system identifyingphysical tissue samples that are delivered to the pathologist (e.g., bythe techniques described herein—that is, by automatically reading orscanning the unique identifier (including, but not limited to, uniquenumerical identifier, unique alphanumeric identifier, unique symbolicidentifier, unique 1D visual code (e.g., bar code, other 1D pattern, orthe like), unique 2D visual code (e.g., QR code, other 2D pattern ormatrix code, or the like), unique 3D visual code (e.g., 3D pattern code,3D matrix code, 3D holographic code, 3D holographic pattern code, 3Dholographic matrix code, or the like), or the like), or a digital imageof at least a portion of the object itself or of a representativeexample of the object (e.g., digital image of at least a portion of atissue sample or tissue section, etc.), on the ID tag of each physicaltissue sample, or by automatically identifying a unique identifyingfeature or attribute of each physical tissue sample, or the like, asdescribed herein). The computing system might then generate and displayimage overlays (and perhaps also generate and present auralinstructions) presenting key information to the pathologist so that thepathologist can be assisted in identifying the most urgent samples to bediagnosed or the most efficient order to diagnose the presented samples.The computing system might also generate image overlays (and perhapsalso generate and present aural instructions) guiding the pathologist toreturn each sample to the proper location (e.g., folder or other samplecontainer, or the like) to ensure that no samples (whether disposed inmicroscope slides or other sample containers, etc.) are misplaced. Insome cases, the image overlays might include use of highlighting fields,holographic arrows, still diagram montages, and/or moving or dynamicdiagrams depicting the key information to the pathologist, the diagnosisprocedures required, the steps for each diagnosis procedure, the orderfor performing the diagnoses, the locations for transfer of samplesbefore, during, and after diagnosis, and so on. Although a particularset of examples is described with respect to performing processes in apathologist's office, the various embodiments are not so limited, andthe AR assistance may be implemented in other ways to perform othertasks in a pathologist's office, consistent with or in accordance withthe various embodiments described herein.

In the case of performing tasks after slide diagnosis, a non-limitingexample might involve the computing system identifying (by thetechniques described herein) each sample or sample container (e.g.,microscope slides containing the sample, or other sample containerscontaining the sample, or the like) after the samples have beendiagnosed by a pathologist. The computing system might generate anddisplay image overlays (and perhaps also generate and present auralinstructions) to users or technicians wearing the AR headsets toretrieve these samples and to return them to the laboratory for storage,by guiding the users or technicians in placing the samples (in theirsample containers) in numerical, alphabetical, or alphanumeric order forlong term storage, as well as assisting the users or technicians insorting (whether in numerical, alphabetical, or alphanumeric order, orthe like) and storing the samples (e.g., tissue blocks or the like) forlong term storage and subsequent (or future) retrieval, as necessary. Insome cases, the image overlays might include use of highlighting fields,holographic arrows, still diagram montages, and/or moving or dynamicdiagrams depicting identification information for each sample (or samplecontainer), information to assist the users or technicians in thesorting tasks, information to assist the users or technicians in thestoring tasks, and/or the like. Although a particular set of examples isdescribed with respect to performing tasks after slide diagnosis, thevarious embodiments are not so limited, and the AR assistance may beimplemented in other ways to perform tasks before or during slidediagnosis, consistent with or in accordance with the various embodimentsdescribed herein.

In the case of performing processes in a clinical or analyticallaboratory, a non-limiting example might involve the computing systemidentifying (by the techniques described herein) a sample (including,but not limited to, a tube of blood, an aliquot from a pharmaceuticalprocess, a field sample, a cell sample, or other sample to be analyzed)that has arrived at a clinical or analytical laboratory. The computingsystem might generate and display image overlays (and perhaps alsogenerate and present aural instructions) to users or technicians wearingthe AR headsets to guide the users or technicians to perform the currentor next appropriate steps for the sample—including, but not limited to,placing the sample in an automated clinical chemistry system, placingthe sample in a centrifuge, placing the sample in a mass spectrometer,or otherwise following a standard or specific protocol for preparing thesample for the next step(s) and for appropriately processing the sample.At each processing step, the computing system might generate and displayimage overlays (and perhaps also generate and present auralinstructions) to guide the users or technicians to the next necessarystep for that specific sample and to supply appropriate sample-specificinformation (that is either learned from the previous step or that isobtained at some other point), if needed. If the sample is split todifferent aliquots or changes in form, the additional tubes or processedsample forms are uniquely identified and those unique identifiers areassociated with the original sample orders. With the generated imageoverlays, the users or technicians wearing the AR headsets can view thelaboratory, and can clearly see or identify which samples are behindschedule, so that those samples can be expedited as their informationviewed in the AR headsets may be tagged with a warning tag, a differentcolor, or other distinguishing characteristics or marks. In someembodiments, the status of instrumentation may also be connected to thedatabase to provide, via the AR headset (and associated image overlaysand/or aural instructions), visual (and/or aural) status feedback to theusers or technicians. Some exemplary (yet non-limiting) informationmight include, without limitation, “run complete,” “run to be finishedin X minutes,” “error,” “reagents required,” “idle” or “standby,” and/orthe like. In some cases, the image overlays might include use ofhighlighting fields, holographic arrows, still diagram montages, and/ormoving or dynamic diagrams depicting the transfer instructions indicatedabove, the next steps, the information regarding the sample, status orwarning information, etc. Although a particular set of examples isdescribed with respect to performing processes in a clinical oranalytical laboratory, the various embodiments are not so limited, andthe AR assistance may be implemented in other ways to perform othertasks in a clinical or analytical laboratory or to perform tasks inother work environments, consistent with or in accordance with thevarious embodiments described herein.

In the case of transferring reagents or samples, a non-limiting examplemight involve the computing system identifying (by the techniquesdescribed herein) all reagents or samples within a workspace (e.g.,counter-top, table, tray, etc.) and/or identifying all transfer tubes,well-plates, or other containers within the workspace, etc. Thecomputing system might generate and display image overlays (and perhapsalso generate and present aural instructions) indicating which reagentsor samples from which set or ones of tubes, well-plates, or othercontainers to transfer to which other set or ones of tubes, well-plates,or other containers. In some cases, different color coding overlays ordifferent highlighting overlays, or the like, may be used to improvevisual distinction between or among the various sets of tubes,well-plates, or other containers corresponding to the different reagentsor samples, or the like. In this manner, the users or technicians, whoare wearing the AR headsets, are better able to more quickly, moreaccurately, more precisely, and with more confidence transfer (in somecases, by pipetting, or by other means of transfer, or the like),reagents or samples into the designated tubes or well-plates (e.g., 96-,384-, or 1536-well plates, or the like) even if the users or techniciansare using multi-well pipettes or the like, while making fewer errors inthe transfer of reagents or samples. In a similar manner, suchAR-assisted transfer of reagents or samples may also be used to loadreagents or samples into laboratory instrumentation or other sampleloading platforms, thereby ensuring that the correct sample(s) istransferred to the correct inlets, sample lanes, or the like. In somecases, the user or technician might use voice input or the like (e.g.,by using a microphone or other audio sensors, such as audio sensor(s)175 of FIG. 1, or the like) to perform one or more of: changing orcorrecting the names of the reagents or samples; entering, changing, orupdating other information about the reagents or samples (e.g.,composition information, storage information, physician information,laboratory information, etc.); changing processes associated withtransfer of the reagents or samples; changing processes associated withdownstream processes involving the reagents or samples; etc. Although aparticular set of examples is described with respect to transfer ofreagents or samples, the various embodiments are not so limited, and theAR assistance may be implemented in other ways to transfer reagents orsamples, consistent with or in accordance with the various embodimentsdescribed herein.

Although FIG. 2 describes the sorting of samples in microscope slides asan example of a task to be performed using AR-based assistance, thevarious embodiments are not so limited, and the AR-based assistance maybe used to sort or otherwise process any or all of a variety oflaboratory materials. For example, in some embodiments, AR-basedassistance may be used for sorting or processing samples in samplecontainers that include, but are not limited to, at least one ofmicroscope slides, cartridges, blocks, vials, tubes, capsules, flasks,vessels, receptacle, holders, repositories, canisters, microarrays, ormicrofluidic chips, and/or the like. In fact, the sample containers tobe sorted or processed may comprise any other suitable type of carrier,container, or vehicle for any of a variety of solid, liquid, and/orgaseous materials, including, without limitation, biological samples(e.g., tissue samples, biomaterial samples, biological specimens, otherbiological samples, genomic samples, pharmaceutical samples, etc.),chemical samples (e.g., pharmaceutical samples, pharmacological samples,drug compounds, reagent samples, organic samples, inorganic samples,other chemical samples, etc.), environmental samples (e.g., soilsamples, rock samples, stone samples, forensic samples, etc.), and/orthe like.

These and other functions or features of the AR-based assistanceimplementation are described in greater detail above or below withrespect to FIGS. 1, 3, and 4.

FIGS. 3A and 3B (collectively, “FIG. 3”) are schematic diagramsillustrating a set of non-limiting examples 300 and 300′ of sorting,categorizing, arranging, or organizing of objects performed by aplurality of users by utilizing AR headsets that are used forimplementing AR-based assistance within a work environment, inaccordance with various embodiments. FIG. 3A depicts a top view of anexample 300 of a portion of a work environment in which sorting,categorizing, arranging, or organizing of objects is performed by aplurality of users 345 who utilize AR headsets 305 for implementingAR-based assistance within a work environment (only two users 345 areshown in FIG. 3 for simplicity of illustration, but the system may beimplemented to accommodate any suitable number of users in anyparticular setting), while FIG. 3B depicts a non-limiting example 300′of a first user's perspective looking through an AR headset that is usedfor implementing AR-based assistance within a work environment asdepicted in FIG. 3A. Sorting of slides by multiple users is described ingreater detail in the '122 Application (which has already beenincorporated herein by reference in its entirety for all purposes), inwhich “artificial reality” refers to “augmented reality” as describedherein. Although the sorting, categorizing, arranging, or organizing ofobjects is shown in FIG. 3 as being a task that may be enhanced by theuse of AR-based assistance or functionality as described herein, thevarious embodiments are not so limited, and the task that may beenhanced by the use of AR-based assistance or functionality mayalternatively include, without limitation, one of running a test sampleor a tissue sample using laboratory instrumentation, solving a puzzle,assembling a piece of machinery, assembling an object, preparing a testsample or a tissue sample, identifying or tracking samples within alaboratory, instructing or guiding users regarding how to performexperimental protocols or tests in a laboratory, performing processes ina histology laboratory, performing processes in a pathologist's office,performing tasks after slide diagnosis, performing processes in aclinical or analytical laboratory, or transferring reagents or samples,and/or the like (not shown), to be performed in corresponding one of alaboratory, a construction site, a machine shop, a workshop, a factory,or a room, and/or the like, where objects of interest within such workenvironments might include, but are not limited to, one of laboratoryinstrumentation, laboratory tools, sample transfer devices, puzzlecomponents or pieces, machinery parts, assembly tools, measurementtools, object parts, sample reagents, sample containers, burners,coolers, mixers, sample preparation tools, or sample transfer tools,and/or the like.

According to some embodiments, examples 300 and 300′ of FIGS. 3A and 3Bmight comprise AR headsets 305 a and 305 b worn by users 345 a and 345 b(respectively), one or more front or front-facing cameras 310 a or 310 b(collectively, “cameras 310” or “image capturing devices 310” or thelike), one or more eye tracking sensors 315 a or 315 b (collectively,“eye tracking sensors 315” or “eye tracking cameras 315” or the like),light sources 375 mounted on each AR headset 305, and/or the like.

In the non-limiting examples 300 and 300′ of FIGS. 3A and 3B, a table orcounter 320 might be disposed within a work environment (in this case, alaboratory, or the like). On the table or counter 320 might be disposeda plurality of microscope slides 325 to be sorted, categorized,arranged, or organized as part of a task to be completed by a user,where each slide 325 might have affixed thereto an identifier oridentification (“ID”) tag 325 a. In the non-limiting example of FIG. 3,each microscope slide 325 might contain a sample 330 for testing,cataloguing, or other laboratory or scientific purpose. As part of thetask associated with the slides 325, each user 345 may be tasked withsorting, categorizing, arranging, or organizing the slides 325, whichmight involve placing particular slides (or sets/groups of slides) inparticular trays 335 or containers 340 (each type of which, in somecases, may be stackable, as shown in FIG. 3B), each tray 335 orcontainer 340 having an ID tag 335 a or 340 a, respectively, affixedthereto. In some cases, the table or counter 320 might also have an IDtag 320 a affixed thereto.

In some instances, the ID tags 320 a, 325 a, 335 a, and/or 340 a mighteach comprise a unique identifier, which might include, withoutlimitation, a unique numerical identifier, a unique alphanumericidentifier, a unique symbolic identifier, a unique one-dimensional(“1D”) visual code (including, but not limited to, a unique bar code, orother unique 1D pattern code, or the like), a unique two-dimensional(“2D”) visual code (including, but not limited to, a unique quickresponse (“QR”) code, or other unique 2D pattern or matrix code, or thelike), a unique three-dimensional (“3D”) visual code (including, but notlimited to, a unique 3D pattern code, a unique 3D matrix code, a unique3D holographic code, a unique 3D holographic pattern code, a unique 3Dholographic matrix code, or the like), a digital image of at least aportion of the object itself or of a representative example of theobject (e.g., digital image of at least a portion of a tissue sample ortissue section, table, tray, container, instrumentation, tool, or otherobject, etc.), or a unique identifying feature or attribute (of anobject among one or more objects 320, 325, 335, 340, etc.) (shown inFIG. 3), and/or the like. Alternatively, or additionally, at least oneof the ID tags 320 a, 325 a, 335 a, and/or 340 a might each comprise aradio frequency identifier (“RFID”) tag, or the like, that wheninterrogated by a radio frequency scanner (which may be disposed in theAR headset 305 (not shown in FIG. 3), or may be disposed within adedicated RFID scanner or other similar device (not shown in FIG. 3))would broadcast identification information (which might include theunique identifier data, or the like) in rf response signals. [Thisdescribes a passive RFID tag, although a battery-assisted passive RFIDtag may be used. An active RFID tag that broadcasts its data withoutneed of power from an interrogating signal from a rf scanner (in somecases, broadcasting constantly) could be used, but within the slidesorting, categorizing, arranging, or organization context, the computingsystem (such as described above with respect to FIG. 1) may be requiredto distinguish amongst the multitude of rf signals from the plurality ofslides. A rf scanner having a narrow field of rf interrogation forinterrogating passive or battery-assisted passive RFID tags within alimited area would avoid the problem inherent with active RFID tags forsuch a task. For other tasks that do not involve many RFID tags within alimited area, active RFID tags may function as well or better thanpassive or battery-assisted passive RFID tags. Alternatively, even fortasks that involve many RFID tags within a limited area (such as theslide sorting, categorizing, arranging, or organization of objects,etc.), the use of gaze focusing techniques as described below (i.e.,tracking what the central portion of the field of view of the user todetermine what the user is looking at) or the use of gestures by theuser's hand(s) (i.e., having the user point directly at a particularobject(s) having an RFID, or the like) or the use of gestures by apointer or selection device(s) operated by the user, and/or the like,may serve as a trigger to filter out (in any generated image overlays)active RFID tag information associated with objects that are outside ofthese areas (i.e., outside of the central portion of the field of viewof the user, outside the area where the user is pointing with the user'shand(s), or outside the area where the user is pointing using thepointer or selection device, etc.).] In some embodiments, a smart tagthat combines RFID tag functionality with printed ID functionality(e.g., with use of barcodes or other 1D visual codes, QR codes or other2D visual codes, 3D visual codes, numbers, letters, text, code, etc.)may also be used. Alternative, or additional, to the use of ID tags 320a, 325 a, 335 a, and/or 340 a for identification of objects (such as thetables or counters 320, slides 325, the trays 335, or the containers340, and/or the like), image recognition or photogrammetric recognitionfunctionality (particularly as enhanced by use of an AI system or thelike) may be used to perform, or aid in the performance of,identification of objects of interest among the objects.

With reference to FIG. 3B, AR-based assistance may be implemented (anon-limiting example 300′ of which is shown in FIG. 3B). For instance,referring to FIG. 3B, a view of a portion of a work environment (in thiscase, a laboratory, or the like) is seen by a first user 345 a throughthe AR headset 305 a. In particular, the cameras 310 a and/or 310 bmight capture an image(s) or video (collectively, “captured images” orthe like) of the portion of the work environment (e.g., laboratory inthis case, or the like). A computing system (such as at least one of theprocessor 150 of the first AR headset 115 a, the computing system 105 a,the remote computing system 105 b, and/or the AI system 140 of FIG. 1,or the like, as described above) might receive the captured images fromthe cameras 310 a and/or 310 b, might analyze the captured images toidentify one or more first objects among the objects present in thecaptured images of the work environment, might query a database (e.g.,data store 155, database(s) 110 a, and/or database(s) 110 b of FIG. 1,or the like) to determine a first task associated with at least onefirst object among the identified one or more first objects, and mightgenerate a first image overlay. In some embodiments, the first imageoverlay might provide at least one of a graphical icon-basedinstruction, a text-based instruction, an image-based instruction, or ahighlighting-based instruction, and/or the like, each indicative of oneor more instructions to be presented to the user to implement the firsttask associated with the at least one first object. The computing systemmight then display, to the eyes of the first user 345 a through the ARheadset 305 a, the generated first image overlay that overlaps with thefield of view of the eyes of the first user 345 a. Turning to FIG. 3B,the first image overlay might include, but is not limited to, a firstseries of instructions 350, a second series of instructions 355, a thirdseries of instructions 360, a fourth series of instructions 365, and/orthe like.

In some embodiments, such as depicted in FIG. 3B, the first series ofinstructions 350 might include, without limitation, a highlighting-basedfield 350 a that, when displayed in the first image overlay, appears tothe user to surround a first slide 325 among the plurality of slides325; an image of an arrow 350 b that points from the first slide 325 toa particular slot within a first container 340 (denoted as “Folder 1” inFIG. 3B); and a hologram or image of a slide 350 c fitting within theparticular slot within first container 340; or the like. In a similarmanner, the second series of instructions 355 might include, but is notlimited to, a highlighting-based field 355 a that covers a second slide325 among the plurality of slides 325; an image of an arrow 355 b thatpoints from the second slide 325 to a particular indentation, cut-out,or bed within a tray 335 (denoted as “Tray 1” in FIG. 3B); and ahighlighting-based field 355 c that covers the particular indentation,cut-out, or bed within the particular tray 335; or the like. Likewise,the third series of instructions 360 might include, without limitation,a highlighting-based field 360 a that, when displayed in the first imageoverlay, appears to the user to surround a third slide 325 (denoted as“Slide 109” in FIG. 3B) among the plurality of slides 325; an image ofan arrow 360 b that points from the third slide 325 to a particular slotwithin a third container 340 (denoted as “Folder 3” in FIG. 3B); and ahighlighting-based field or image 360 c outlining the bottom portion ofthe particular slot within third container 340; or the like. Similarly,the fourth series of instructions 365 might include, but is not limitedto, a highlighting-based field 365 a that covers a fourth slide 325among the plurality of slides 325; an image of an arrow 365 b thatpoints from the fourth slide 325 to a particular indentation, cut-out,or bed within a tray 335 (denoted as “Tray 1” in FIG. 3B); and ahighlighting-based field 365 c that covers the particular indentation,cut-out, or bed within the particular tray 335; or the like. The firstand second series of instructions 350 and 355 are intended forinstructing the first user 345 a to perform one or more tasks, while thethird and fourth series of instructions 360 and 365 are intended forinstructing the second user 345 b to perform one or more other tasks. Asshown in FIG. 3B, the first and second series of instructions 350 and355 are depicted with a slash-type pattern, thereby distinguishing themfrom the third and fourth series of instructions 360 and 365, which aredepicted with a textured pattern.

In some embodiments, rather than instructions, information may bedisplayed; for purposes of simplicity of description, however, suchinformation is also referred to herein as instructions. For example,text-based tags 370 may be displayed, as an image overlay, beside oraround some or all of relevant objects within the work environment. Thetext-based tags 370 might include, for instance, a slide-identifyingtext-based tag 370 a (e.g., identifying “Slide 109” or the like), acontainer-identifying text-based tag 370 b (e.g., identifying “Folder 1”or the like), another container-identifying text-based tag 370 c (e.g.,identifying “Folder 3” or the like), and a tray-identifying text-basedtag 370 d (e.g., identifying “Tray 1” or the like). According to someembodiments, the light source 375 might be used to highlight (using acollimated or focused beam of light) a focused portion of the workenvironment (such as a point at which the user 345 is looking at (whichwould serve as a cursor, reticle, or the like) or a point that thecomputing system highlights for the user to focus in on, or the like).Although particular examples of image overlays are illustrated in FIG.3B, the various embodiments are not so limited, and any suitable type,shape, configuration, orientation, color, or size of image overlays maybe used consistent with the embodiments described herein, with differenttypes, shapes, configurations, orientations, colors, and/or sizes beingdisplayed to represent responsibility by, or assignment to, thedifferent users.

For multi-user slide sorting, the following four scenarios may arise,particularly, in the case of a two user situation: (1) each user has hisor her own sets of slides and his or her own piles of folders; (2) eachuser has his or her own sets of slides, but work on the same piles offolders; (3) the users share common sets of slides and common piles offolders; and (4) the users share common sets of slides, but each haveindividual piles of folders; and/or the like. For situations involvingmore than two users, modifications may be made to these scenarios.

In the first scenario, each user sorts his, her, or their current pileinto a single empty folder until a new folder is needed. If the user nowhas remaining slides that could go into another user's folder pile, thatuser is instructed to move those slides to the other user's loose slideregion. Users continue to add new folders to their own pile(s) and tosort slides in their piles to their own folders. Slides that could besorted into another user's pile are directed to be moved to theappropriate user's loose slide regions. If another user's top folder iswithin reach and the user has slides that can be placed there, the usercan be instructed to place the slides in the other user's folder. Theother user should be instructed to not add a new folder until theseslides are placed. Alternatively, the user may be instructed to simplymove the slides to the other user's loose slide region for the otheruser to sort. As each user adds more slides to his, her, or their slideregion, each user is directed to add or move his, her, or their ownfolders around as appropriate, and to move slides to other user'sregions as needed.

In the second scenario, on starting to sort, each user is instructed toadd a new folder to different locations on the table. In the even thatall slides from both users would go to a single folder, only one user isinstructed to add a folder. Each user sorts from his, her, or theirslides into any of the top folders. If both users have no more slidesfor any top folder, one or both users are instructed to add a new folderto the table, either on top of an existing folder, or in a new locationdepending on available space. All folder piles should be in reach ofboth users. When or if this is no longer possible, the user that hasslides that cannot be placed within reach is instructed to move theappropriate slides to the other user's slide region (assuming that iswithin reach).

In the third scenario, the users would work in close proximity to eachother. In the fourth scenario, each user might be instructed to addparticular slides to his, her, or their own folder, while moving otherslides to the other user's portion of the slide region.

Although FIG. 3, like FIG. 2, describes the sorting of samples inmicroscope slides as an example of a task to be performed using AR-basedassistance, the various embodiments are not so limited, and the AR-basedassistance may be used to sort or otherwise process any or all of avariety of laboratory materials. For example, in some embodiments,AR-based assistance may be used for sorting or processing samples insample containers that include, but are not limited to, at least one ofmicroscope slides, cartridges, blocks, vials, tubes, capsules, flasks,vessels, receptacle, holders, repositories, canisters, microarrays, ormicrofluidic chips, and/or the like. In fact, the sample containers tobe sorted or processed may comprise any other suitable type of carrier,container, or vehicle for any of a variety of solid, liquid, and/orgaseous materials, including, without limitation, biological samples(e.g., tissue samples, biomaterial samples, biological specimens, otherbiological samples, genomic samples, pharmaceutical samples, etc.),chemical samples (e.g., pharmaceutical samples, pharmacological samples,drug compounds, reagent samples, organic samples, inorganic samples,other chemical samples, etc.), environmental samples (e.g., soilsamples, rock samples, stone samples, forensic samples, etc.), and/orthe like.

The AR-based assistance implementation and/or the image overlays of FIG.3 might otherwise be similar, or identical, to the AR-based assistanceimplementation and/or the image overlays of FIGS. 1 and 2, or the like.These and other functions or features of the AR-based assistanceimplementation are described in greater detail above or below withrespect to FIGS. 1, 2, and 4.

FIGS. 4A and 4B (collectively, “FIG. 4”) are flow diagrams illustratinga method for implementing AR-based assistance within a work environment,in accordance with various embodiments. Method 400 of FIG. 4A continuesonto FIG. 4B following the circular marker denoted, “A.”

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method 400 illustrated byFIG. 4 can be implemented by or with (and, in some cases, are describedbelow with respect to) the systems, examples, or embodiments 100, 200,200′, 300, and 300′ of FIGS. 1, 2A, 2B, 3A, and 3B, respectively (orcomponents thereof), such methods may also be implemented using anysuitable hardware (or software) implementation. Similarly, while each ofthe systems, examples, or embodiments 100, 200, 200′, 300, and 300′ ofFIGS. 1, 2A, 2B, 3A, and 3B, respectively (or components thereof), canoperate according to the method 400 illustrated by FIG. 4 (e.g., byexecuting instructions embodied on a computer readable medium), thesystems, examples, or embodiments 100, 200, 200′, 300, and 300′ of FIGS.1, 2A, 2B, 3A, and 3B can each also operate according to other modes ofoperation and/or perform other suitable procedures.

In the non-limiting embodiment of FIG. 4A, method 400, at block 405,might comprise receiving, with a computing system and from one or morefirst image capture devices having a field of view of at least a firstportion of a work environment (i.e., directed in front of an augmentedreality (“AR”) headset worn by a first user, or the like), one or morefirst images of the at least a first portion of the work environment,the one or more first images overlapping with a field of view of eyes ofthe first user wearing the AR headset. According to some embodiments,the computing system might include, without limitation, one of aprocessor disposed in the AR headset, a computing system disposed in thework environment, a remote computing system disposed external to thework environment and accessible over a network, or a cloud computingsystem, and/or the like. In some cases, the AR headset might include,but is not limited to, one of a set of AR goggles, a pair of AR-enabledeyewear, an AR-enabled smartphone mounted in a headset, or an AR helmet,and/or the like. In some instances, at least one of the one or morefirst image capture devices might be disposed on a portion of a housingof the AR headset.

At optional block 410, method 400 might comprise receiving, with thecomputing system and from one or more second image capture devices thatare disposed within the work environment but external to the AR headset,one or more second images of at least a second portion of the workenvironment.

Method 400 might further comprise analyzing, with the computing system,the received one or more first images to identify one or more firstobjects disposed in the work environment (block 415). Method 400 mightfurther comprise, at optional block 420, analyzing, with the computingsystem, the received one or more first images and the received one ormore second images to identify the one or more first objects disposed inthe work environment. At block 425, method 400 might comprise queryingat least one database, with the computing system, to determine a firsttask associated with at least one first object among the identified oneor more first objects. In some embodiments, the first task mightinclude, without limitation, sorting, categorizing, arranging, ororganizing the one or more first objects, wherein the one or moreobjects might include, but is not limited to, one or more microscopeslides each containing a test sample or a tissue sample. In some cases,a plurality of users are tasked with performing the first task, whereineach user wears an AR headset that displays generated image overlays toeach user that distinguishes objects that one user is intended tointeract with from objects that the other users among the plurality ofusers are intended to interact with as part of the first task.Alternatively, the first task might include, without limitation, one ofrunning a test sample or a tissue sample using laboratoryinstrumentation; assembling a piece of machinery; assembling an object;preparing a test sample or a tissue sample; identifying and/or trackingsamples within a laboratory; instructing or guiding users regarding howto perform experimental protocols or tests in a laboratory; performingprocesses in a histology laboratory; performing processes in apathologist's office; performing tasks after slide diagnosis; performingprocesses in a clinical or analytical laboratory; or transferringreagents or samples; and/or the like. In some cases, the one or morefirst objects might include, but is not limited to, one of laboratoryinstrumentation, laboratory tools, sample transfer devices, puzzlecomponents or pieces, machinery parts, assembly tools, measurementtools, object parts, sample reagents, sample containers, burners,coolers, mixers, sample preparation tools, or sample transfer tools,and/or the like. In some cases, the work environment comprises one of alaboratory, a construction site, a machine shop, a workshop, a factory,or a room, and/or the like.

Method 400, at block 430, might comprise generating, with the computingsystem, a first image overlay, the first image overlay providing one ormore instructions (which might include, without limitation, at least oneof a graphical icon-based instruction, a text-based instruction, animage-based instruction, or a highlighting-based instruction, and/or thelike) to be presented to a user to implement the first task associatedwith the at least one first object.

In some instances, the at least one of the graphical icon-basedinstruction, the text-based instruction, the image-based instruction, orthe highlighting-based instruction might include, but is not limited to,at least one of a graphical icon-based instruction, a text-basedinstruction, an image-based instruction, or a highlighting-basedinstruction that, when displayed within the generated first imageoverlay, is superimposed over, displayed around, or displayed beside theat least one first object as viewed by the user through the AR headset,and/or the like. In some cases, the graphical icon-based instructionmight include, without limitation, at least one of a graphical iconrepresenting identification information associated with the at least onefirst object; a graphical icon representing identification informationassociated with a second object with which the at least one first objectis intended to interact as part of the first task; a colored graphicalicon distinguishing one first object from another first object among theat least one first object; a graphical icon comprising a directionalarrow representing a direction that the at least one first object shouldbe taken as part of the first task; or a colored graphical iconcomprising a colored directional arrow distinguishing a first directionthat the at least one first object should be taken as part of the firsttask from a second direction that a third object should be taken as partof the first task, and/or the like. In some instances, the text-basedinstruction might include, but is not limited to, at least one of afloating text window comprising textual instructions corresponding tothe first task; or a surface text window that is superimposed on one ofa table-top surface, a wall surface, or an object surface and thatcomprises textual instructions corresponding to the first task. In somecases, the floating text window, when displayed within the generatedfirst image overlay, is displayed as a floating image beside the atleast one first object or displayed as a floating image within the fieldof view of the eyes of the first user.

In some instances, the image-based instruction might include, withoutlimitation, at least one of an image representing identificationinformation associated with the at least one first object; an imagerepresenting identification information associated with a second objectwith which the at least one first object is intended to interact as partof the first task; an image comprising a directional arrow representinga direction that the at least one first object should be taken as partof the first task; an image comprising images of numbers or codesrepresenting an order of processes of the first task associated with theat least one first object; a magnified image of the at least one firstobject; a three-dimensional (“3D”) image or hologram; or an image of theat least one first object superimposed over a targeted portion of thework environment indicating at least one of position, orientation, orconfiguration of the at least one first object intended for placement atthe targeted portion; and/or the like. In some cases, thehighlighting-based instruction might include, but is not limited to, atleast one of a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over the at least onefirst object; a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a fourth object thatis related to the first task associated with the at least one firstobject; or a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a targeted portionof the work environment indicated intended placement of the at least onefirst object at the targeted portion; and/or the like.

In some embodiments, the AR headset might include, without limitation,one or more of at least one earpiece or at least one speaker. In suchcases, method 400 might further comprise generating, with the computingsystem, one or more audio-based instructions (which might include, butis not limited to, at least one of a tone, a series of tones, spokeninstructions, or test-to-speech content, and/or the like) correspondingto each of at least one of the one or more instructions to be presentedto the user via the generated first image overlay (optional block 435).

Method 400 might further comprise, at block 440, displaying, with thecomputing system and to the eyes of the first user through the ARheadset, the generated first image overlay that overlaps with the fieldof view of the eyes of the first user. According to some embodiments,displaying the generated first image overlay to the eyes of the firstuser through the AR headset might include, but is not limited to, one ofprojecting the generated first image overlay directly in the eyes of thefirst user, projecting the generated first image overlay on a projectionplane or surface in front of the eyes of the first user, projecting thegenerated first image overlay as a hologram in front of the eyes of thefirst user, displaying the generated first image overlay on atransparent or semi-transparent display screen of the AR headset that isdisposed in front of the eyes of the first user, or displaying thegenerated first image overlay superimposed over a continuous videorecording and display on a display screen of an AR-enabled smartphonemounted in the AR headset, and/or the like. Method 400 might continueonto the process at optional block 445 in FIG. 4B following the circularmarker denoted, “A.”

At optional block 445 in FIG. 4B (following the circular marker denoted,“A”), method 400 might comprise presenting, with the computing systemand via one of the one or more of the at least one earpiece or the atleast one speaker, the generated one or more audio-based instructions inconjunction with displaying corresponding each of at least one of theone or more instructions being presented to the user via the generatedfirst image overlay as displayed to the eyes of the first user throughthe AR headset.

Method 400 might further comprise, at optional block 450, illuminating,with a first light source, one or more of the at least one first objector the at least a first portion of the work environment during at leastone of the following periods: prior to the one or more first imagesbeing captured by the one or more first image capture device; or inconjunction with the display of the generated first image overlay. Insome cases, the first light source might be disposed on a portion of theAR headset.

Method 400 might comprise, at optional block 455, tracking, with thecomputing system and using one or more second image capture devices thatare disposed on the AR headset and that have a field of view thatincludes the eyes of the first user (i.e., that are directed at the eyesof the first user, or the like), the eyes of the first user to determinea direction in which the first user is looking. Method 400 might furthercomprise determining, with the computing system, a central portion of afield of view of the first user (i.e., determining what the first useris looking at, or the like) based at least in part on the determineddirection that the first user is looking in conjunction with the one ormore first images as captured by the one or more first image capturedevices (optional 460). At optional block 465, method 400 mightcomprise, in response to determining the central portion of the field ofview of the first user (i.e., determining what the first user is lookingat, or the like), identifying, with the computing system, at least onefifth object.

Exemplary System and Hardware Implementation

FIG. 5 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments. FIG. 5provides a schematic illustration of one embodiment of a computer system500 of the service provider system hardware that can perform the methodsprovided by various other embodiments, as described herein, and/or canperform the functions of computer or hardware system (i.e., computingsystems 105 a and 105 b, augmented reality (“AR”) headsets 115 a-115 n,205, 305 a, and 305 b, and artificial intelligence (“AI”) system 140,etc.), as described above. It should be noted that FIG. 5 is meant onlyto provide a generalized illustration of various components, of whichone or more (or none) of each may be utilized as appropriate. FIG. 5,therefore, broadly illustrates how individual system elements may beimplemented in a relatively separated or relatively more integratedmanner.

The computer or hardware system 500—which might represent an embodimentof the computer or hardware system (i.e., computing systems 105 a and105 b, AR headsets 115 a-115 n, 205, 305 a, and 305 b, and AI system140, etc.), described above with respect to FIGS. 1-4—is showncomprising hardware elements that can be electrically coupled via a bus505 (or may otherwise be in communication, as appropriate). The hardwareelements may include one or more processors 510, including, withoutlimitation, one or more general-purpose processors and/or one or morespecial-purpose processors (such as microprocessors, digital signalprocessing chips, graphics acceleration processors, and/or the like);one or more input devices 515, which can include, without limitation, amouse, a keyboard, and/or the like; and one or more output devices 520,which can include, without limitation, a display device, a printer,and/or the like.

The computer or hardware system 500 may further include (and/or be incommunication with) one or more storage devices 525, which can comprise,without limitation, local and/or network accessible storage, and/or caninclude, without limitation, a disk drive, a drive array, an opticalstorage device, solid-state storage device such as a random accessmemory (“RAM”) and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable, and/or the like. Such storage devicesmay be configured to implement any appropriate data stores, including,without limitation, various file systems, database structures, and/orthe like.

The computer or hardware system 500 might also include a communicationssubsystem 530, which can include, without limitation, a modem, a networkcard (wireless or wired), an infra-red communication device, a wirelesscommunication device and/or chipset (such as a Bluetooth™ device, an802.11 device, a WiFi device, a WiMax device, a WWAN device, cellularcommunication facilities, etc.), and/or the like. The communicationssubsystem 530 may permit data to be exchanged with a network (such asthe network described below, to name one example), with other computeror hardware systems, and/or with any other devices described herein. Inmany embodiments, the computer or hardware system 500 will furthercomprise a working memory 535, which can include a RAM or ROM device, asdescribed above.

The computer or hardware system 500 also may comprise software elements,shown as being currently located within the working memory 535,including an operating system 540, device drivers, executable libraries,and/or other code, such as one or more application programs 545, whichmay comprise computer programs provided by various embodiments(including, without limitation, hypervisors, VMs, and the like), and/ormay be designed to implement methods, and/or configure systems, providedby other embodiments, as described herein. Merely by way of example, oneor more procedures described with respect to the method(s) discussedabove might be implemented as code and/or instructions executable by acomputer (and/or a processor within a computer); in an aspect, then,such code and/or instructions can be used to configure and/or adapt ageneral purpose computer (or other device) to perform one or moreoperations in accordance with the described methods.

A set of these instructions and/or code might be encoded and/or storedon a non-transitory computer readable storage medium, such as thestorage device(s) 525 described above. In some cases, the storage mediummight be incorporated within a computer system, such as the system 500.In other embodiments, the storage medium might be separate from acomputer system (i.e., a removable medium, such as a compact disc,etc.), and/or provided in an installation package, such that the storagemedium can be used to program, configure, and/or adapt a general purposecomputer with the instructions/code stored thereon. These instructionsmight take the form of executable code, which is executable by thecomputer or hardware system 500 and/or might take the form of sourceand/or installable code, which, upon compilation and/or installation onthe computer or hardware system 500 (e.g., using any of a variety ofgenerally available compilers, installation programs,compression/decompression utilities, etc.) then takes the form ofexecutable code.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware (such as programmable logic controllers,field-programmable gate arrays, application-specific integratedcircuits, and/or the like) might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer or hardware system (such as the computer or hardware system500) to perform methods in accordance with various embodiments of theinvention. According to a set of embodiments, some or all of theprocedures of such methods are performed by the computer or hardwaresystem 500 in response to processor 510 executing one or more sequencesof one or more instructions (which might be incorporated into theoperating system 540 and/or other code, such as an application program545) contained in the working memory 535. Such instructions may be readinto the working memory 535 from another computer readable medium, suchas one or more of the storage device(s) 525. Merely by way of example,execution of the sequences of instructions contained in the workingmemory 535 might cause the processor(s) 510 to perform one or moreprocedures of the methods described herein.

The terms “machine readable medium” and “computer readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In an embodimentimplemented using the computer or hardware system 500, various computerreadable media might be involved in providing instructions/code toprocessor(s) 510 for execution and/or might be used to store and/orcarry such instructions/code (e.g., as signals). In manyimplementations, a computer readable medium is a non-transitory,physical, and/or tangible storage medium. In some embodiments, acomputer readable medium may take many forms, including, but not limitedto, non-volatile media, volatile media, or the like. Non-volatile mediaincludes, for example, optical and/or magnetic disks, such as thestorage device(s) 525. Volatile media includes, without limitation,dynamic memory, such as the working memory 535. In some alternativeembodiments, a computer readable medium may take the form oftransmission media, which includes, without limitation, coaxial cables,copper wire, and fiber optics, including the wires that comprise the bus505, as well as the various components of the communication subsystem530 (and/or the media by which the communications subsystem 530 providescommunication with other devices). In an alternative set of embodiments,transmission media can also take the form of waves (including withoutlimitation radio, acoustic, and/or light waves, such as those generatedduring radio-wave and infra-red data communications).

Common forms of physical and/or tangible computer readable mediainclude, for example, a floppy disk, a flexible disk, a hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punch cards, paper tape, any other physical medium with patternsof holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chipor cartridge, a carrier wave as described hereinafter, or any othermedium from which a computer can read instructions and/or code.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 510for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by the computer or hardware system 500. Thesesignals, which might be in the form of electromagnetic signals, acousticsignals, optical signals, and/or the like, are all examples of carrierwaves on which instructions can be encoded, in accordance with variousembodiments of the invention.

The communications subsystem 530 (and/or components thereof) generallywill receive the signals, and the bus 505 then might carry the signals(and/or the data, instructions, etc. carried by the signals) to theworking memory 535, from which the processor(s) 505 retrieves andexecutes the instructions. The instructions received by the workingmemory 535 may optionally be stored on a storage device 525 eitherbefore or after execution by the processor(s) 510.

As noted above, a set of embodiments comprises methods and systems forimplementing assistance within a work environment, and, moreparticularly, to methods, systems, and apparatuses for implementingaugmented reality (AR)-based assistance within a work environment. FIG.6 illustrates a schematic diagram of a system 600 that can be used inaccordance with one set of embodiments. The system 600 can include oneor more user computers, user devices, or customer devices 605. A usercomputer, user device, or customer device 605 can be a general purposepersonal computer (including, merely by way of example, desktopcomputers, tablet computers, laptop computers, handheld computers, andthe like, running any appropriate operating system, several of which areavailable from vendors such as Apple, Microsoft Corp., and the like),cloud computing devices, a server(s), and/or a workstation computer(s)running any of a variety of commercially-available UNIX™ or UNIX-likeoperating systems. A user computer, user device, or customer device 605can also have any of a variety of applications, including one or moreapplications configured to perform methods provided by variousembodiments (as described above, for example), as well as one or moreoffice applications, database client and/or server applications, and/orweb browser applications. Alternatively, a user computer, user device,or customer device 605 can be any other electronic device, such as athin-client computer, Internet-enabled mobile telephone, and/or personaldigital assistant, capable of communicating via a network (e.g., thenetwork(s) 610 described below) and/or of displaying and navigating webpages or other types of electronic documents. Although the exemplarysystem 600 is shown with two user computers, user devices, or customerdevices 605, any number of user computers, user devices, or customerdevices can be supported.

Certain embodiments operate in a networked environment, which caninclude a network(s) 610. The network(s) 610 can be any type of networkfamiliar to those skilled in the art that can support datacommunications using any of a variety of commercially-available (and/orfree or proprietary) protocols, including, without limitation, TCP/IP,SNA™, IPX™, AppleTalk™, and the like. Merely by way of example, thenetwork(s) 610 (similar to network(s) 145 FIG. 1, or the like) can eachinclude a local area network (“LAN”), including, without limitation, afiber network, an Ethernet network, a Token-Ring™ network, and/or thelike; a wide-area network (“WAN”); a wireless wide area network(“WWAN”); a virtual network, such as a virtual private network (“VPN”);the Internet; an intranet; an extranet; a public switched telephonenetwork (“PSTN”); an infra-red network; a wireless network, including,without limitation, a network operating under any of the IEEE 802.11suite of protocols, the Bluetooth™ protocol known in the art, and/or anyother wireless protocol; and/or any combination of these and/or othernetworks. In a particular embodiment, the network might include anaccess network of the service provider (e.g., an Internet serviceprovider (“ISP”)). In another embodiment, the network might include acore network of the service provider, and/or the Internet.

Embodiments can also include one or more server computers 615. Each ofthe server computers 615 may be configured with an operating system,including, without limitation, any of those discussed above, as well asany commercially (or freely) available server operating systems. Each ofthe servers 615 may also be running one or more applications, which canbe configured to provide services to one or more clients 605 and/orother servers 615.

Merely by way of example, one of the servers 615 might be a data server,a web server, a cloud computing device(s), or the like, as describedabove. The data server might include (or be in communication with) a webserver, which can be used, merely by way of example, to process requestsfor web pages or other electronic documents from user computers 605. Theweb server can also run a variety of server applications, including HTTPservers, FTP servers, CGI servers, database servers, Java servers, andthe like. In some embodiments of the invention, the web server may beconfigured to serve web pages that can be operated within a web browseron one or more of the user computers 605 to perform methods of theinvention.

The server computers 615, in some embodiments, might include one or moreapplication servers, which can be configured with one or moreapplications accessible by a client running on one or more of the clientcomputers 605 and/or other servers 615. Merely by way of example, theserver(s) 615 can be one or more general purpose computers capable ofexecuting programs or scripts in response to the user computers 605and/or other servers 615, including, without limitation, webapplications (which might, in some cases, be configured to performmethods provided by various embodiments). Merely by way of example, aweb application can be implemented as one or more scripts or programswritten in any suitable programming language, such as Java™, C, C#™ orC++, and/or any scripting language, such as Perl, Python, or TCL, aswell as combinations of any programming and/or scripting languages. Theapplication server(s) can also include database servers, including,without limitation, those commercially available from Oracle™,Microsoft™, Sybase™, IBM™, and the like, which can process requests fromclients (including, depending on the configuration, dedicated databaseclients, API clients, web browsers, etc.) running on a user computer,user device, or customer device 605 and/or another server 615. In someembodiments, an application server can perform one or more of theprocesses for implementing assistance within a work environment, and,more particularly, to methods, systems, and apparatuses for implementingaugmented reality (AR)-based assistance within a work environment, asdescribed in detail above. Data provided by an application server may beformatted as one or more web pages (comprising HTML, JavaScript, etc.,for example) and/or may be forwarded to a user computer 605 via a webserver (as described above, for example). Similarly, a web server mightreceive web page requests and/or input data from a user computer 605and/or forward the web page requests and/or input data to an applicationserver. In some cases, a web server may be integrated with anapplication server.

In accordance with further embodiments, one or more servers 615 canfunction as a file server and/or can include one or more of the files(e.g., application code, data files, etc.) necessary to implementvarious disclosed methods, incorporated by an application running on auser computer 605 and/or another server 615. Alternatively, as thoseskilled in the art will appreciate, a file server can include allnecessary files, allowing such an application to be invoked remotely bya user computer, user device, or customer device 605 and/or server 615.

It should be noted that the functions described with respect to variousservers herein (e.g., application server, database server, web server,file server, etc.) can be performed by a single server and/or aplurality of specialized servers, depending on implementation-specificneeds and parameters.

In certain embodiments, the system can include one or more databases 620a-620 n (collectively, “databases 620”). The location of each of thedatabases 620 is discretionary: merely by way of example, a database 620a might reside on a storage medium local to (and/or resident in) aserver 615 a (and/or a user computer, user device, or customer device605). Alternatively, a database 620 n can be remote from any or all ofthe computers 605, 615, so long as it can be in communication (e.g., viathe network 610) with one or more of these. In a particular set ofembodiments, a database 620 can reside in a storage-area network (“SAN”)familiar to those skilled in the art. (Likewise, any necessary files forperforming the functions attributed to the computers 605, 615 can bestored locally on the respective computer and/or remotely, asappropriate.) In one set of embodiments, the database 620 can be arelational database, such as an Oracle database, that is adapted tostore, update, and retrieve data in response to SQL-formatted commands.The database might be controlled and/or maintained by a database server,as described above, for example.

According to some embodiments, system 600 might further comprisecomputing system 625 and corresponding database(s) 630 (similar tocomputing system 105 a and corresponding database(s) 110 a of FIG. 1, orthe like), one or more augmented reality (“AR”) headsets 635 a-635 n(collectively, “AR headsets 635” or the like; similar to AR headsets 115a-115 n, 205, 305 a, and 305 b of FIGS. 1-3, or the like) that are wornor wearable by one or more users 640 a-640 n (collectively, “users 640”or the like; similar to users 120 a-120 n, 345 a, or 345 b of FIGS. 1and 3A, or the like). In some cases, the AR headset might include, butis not limited to, one of a set of AR goggles, a pair of AR-enabledeyewear, an AR-enabled smartphone mounted in a headset, or an AR helmet,and/or the like. In some embodiments, system 600 might comprise one ormore objects 645 a-645 n (collectively, “objects 645” or the like;similar to objects 125 a-125 n, 220, 225, 235, 240, 320, 325, 335, and340 of FIGS. 1-3, or the like); in some cases, objects 645 mightcomprise identifier or identification (“ID”) tags 650 (optional; similarto ID tags 130 a-130 n, 220 a, 225 a, 235 a, 240 a, 320 a, 325 a, 335 a,and 340 a of FIGS. 1-3, or the like) affixed thereto (the ID tags 650comprising tags 650 a-650 n, or the like, each affixed to one of theobjects 645). In some instances, the ID tags 650 might comprise a uniqueidentifier, which might include, without limitation, a unique numericalidentifier, a unique alphanumeric identifier, a unique symbolicidentifier, a unique one-dimensional (“1D”) visual code (including, butnot limited to, a unique bar code, or other unique 1D pattern code, orthe like), a unique two-dimensional (“2D”) visual code (including, butnot limited to, a unique quick response (“QR”) code, or other unique 2Dpattern or matrix code, or the like), a unique three-dimensional (“3D”)visual code (including, but not limited to, a unique 3D pattern code, aunique 3D matrix code, a unique 3D holographic code, a unique 3Dholographic pattern code, a unique 3D holographic matrix code, or thelike), a digital image of at least a portion of the object 645 itself orof a representative example of the object 645 (e.g., digital image of atleast a portion of a tissue sample or tissue section, table, tray,container, instrumentation, tool, or other object, etc.), or a uniqueidentifying feature or attribute (of an object among the one or moreobjects 645), and/or the like. Alternatively, or additionally, at leastone of the ID tags 650 might comprise a radio frequency identifier(“RFID”) tag, or the like, that when interrogated by a radio frequencyscanner (which may be disposed in the AR headset 635, such astransceiver 190 of FIG. 1 or other component (not shown), or may bedisposed within a dedicated RFID scanner or other similar device (notshown)) would broadcast identification information (which might includethe unique identifier data, or the like) in rf response signals. In someembodiments, a smart tag that combines RFID tag functionality withprinted ID functionality (e.g., with use of barcodes or other 1D visualcodes, QR codes or other 2D visual codes, 3D visual codes, numbers,letters, text, code, etc.) may also be used. Alternative, or additional,to the ID tags 650, image recognition or photogrammetric recognitionfunctionality (particularly as enhanced by use of AI system 670 or thelike) may be used to perform, or aid in the performance of,identification of objects of interest among the objects 645. Thecomputing system 625, the database(s) 630, the AR goggles 635 a-635 nthat are worn or wearable by respective users 640 a-640 n, and theobjects 645 a-645 n, or the like, might be disposed in work environment655, which might include, but is not limited to, one of a laboratory, aconstruction site, a machine shop, a workshop, a factory, or a room,and/or the like.

System 600 might further comprise an artificial intelligence (“AI”)system 670 (optional; similar to AI system 140 of FIG. 1, or the like)that might communicatively couple to computing system 625 via network(s)610. According to some embodiments, alternative or additional to thecomputing system 625 and corresponding database 630 being disposedwithin work environment 655, system 600 might comprise remote computingsystem 660 (optional; similar to remote computing system 105 b of FIG.1, or the like) and corresponding database(s) 665 (optional; similar todatabase(s) 110 b of FIG. 1, or the like) that communicatively couplewith at least one of the one or more AR headsets 635 a-635 n in the workenvironment 655 via the one or more networks 610. According to someembodiments, computing system 625 might include, without limitation, oneof a processor disposed in the AR headset, or a computing systemdisposed in the work environment, and/or the like. In some cases, remotecomputing system 660 might include, but is not limited to, at least oneof a remote computing system disposed external to the work environmentand accessible over a network, or a cloud computing system, and/or thelike. The AI system 670 might be used to facilitate operation bycomputing system 625, computing system 660, and/or at least one ARheadset 635.

In operation, at least one of the front camera(s) (e.g., front camera(s)180 of FIG. 1, or the like) of a first AR headset 635 a and/or the frontcamera(s) of one or more other AR headsets 635 b-635 n (collectively,“image capture device(s)” or “camera(s)” or the like) might captureimages or video of at least a first portion of work environment 655. Insome cases, (particularly, with the images or videos captured by thefront camera(s) of the first AR headset 635 a) the captured images orvideo might overlap with a field of view of the eyes of the first user640 a wearing the first AR headset 635 a. The image capture device(s)might subsequently send the captured images or video. At least one ofthe first AR headset 635 a, the computing system 625, the remotecomputing system 660, and/or the AI system 670 (collectively, “thecomputing system” or the like) might receive the captured images orvideo (collectively, “captured images” or the like) from the imagecapture device(s), might analyze the received captured images toidentify one or more first objects 645 among the one or more objects 645a-645 n, might query at least one database (e.g., database(s) 630 and/ordatabase(s) 665, or the like) to determine a first task associated withat least one first object among the identified one or more first objects645, and might generate a first image overlay. In some embodiments, thefirst image overlay might provide at least one of a graphical icon-basedinstruction, a text-based instruction, an image-based instruction, or ahighlighting-based instruction, and/or the like, each indicative of oneor more instructions to be presented to first user 640 a to implementthe first task associated with the at least one first object 645. Thecomputing system might then display, to the eyes of the first user 640 athrough the AR headset, the generated first image overlay that overlapswith the field of view of the eyes of the first user 640 a.

These and other functions of the system 600 (and its components) aredescribed in greater detail above with respect to FIGS. 1-4.

While certain features and aspects have been described with respect toexemplary embodiments, one skilled in the art will recognize thatnumerous modifications are possible. For example, the methods andprocesses described herein may be implemented using hardware components,software components, and/or any combination thereof. Further, whilevarious methods and processes described herein may be described withrespect to particular structural and/or functional components for easeof description, methods provided by various embodiments are not limitedto any particular structural and/or functional architecture but insteadcan be implemented on any suitable hardware, firmware and/or softwareconfiguration. Similarly, while certain functionality is ascribed tocertain system components, unless the context dictates otherwise, thisfunctionality can be distributed among various other system componentsin accordance with the several embodiments.

Moreover, while the procedures of the methods and processes describedherein are described in a particular order for ease of description,unless the context dictates otherwise, various procedures may bereordered, added, and/or omitted in accordance with various embodiments.Moreover, the procedures described with respect to one method or processmay be incorporated within other described methods or processes;likewise, system components described according to a particularstructural architecture and/or with respect to one system may beorganized in alternative structural architectures and/or incorporatedwithin other described systems. Hence, while various embodiments aredescribed with—or without—certain features for ease of description andto illustrate exemplary aspects of those embodiments, the variouscomponents and/or features described herein with respect to a particularembodiment can be substituted, added and/or subtracted from among otherdescribed embodiments, unless the context dictates otherwise.Consequently, although several exemplary embodiments are describedabove, it will be appreciated that the invention is intended to coverall modifications and equivalents within the scope of the followingclaims.

What is claimed is:
 1. A method, comprising: receiving, with a computingsystem and from one or more first image capture devices having a fieldof view of at least a first portion of a work environment, one or morefirst images of the at least the first portion of the work environment,the one or more first images overlapping with a field of view of eyes ofa first user wearing an augmented reality (“AR”) headset; analyzing,with the computing system, the received one or more first images toidentify one or more first objects disposed in the work environment,wherein the one or more first objects each comprises a radio frequencyidentifier (“RFID”) tag affixed thereto, and wherein to identify the oneor more first objects comprises: identifying the one or more firstobjects based on the RFID tag affixed to each first object inconjunction with identifying the one or more first objects based onanalysis of the received one or more first images; and filtering, basedon a gesture or a selection by the first user, RFID tag informationassociated with objects outside of a central portion of the field ofview of the first user or outside of an area defined by the gesture orthe selection by the first user; querying at least one database, withthe computing system, to determine a first task associated with at leastone first object among the identified one or more first objects, whereinthe first task comprises sorting, categorizing, arranging, or organizingthe one or more first objects, wherein the one or more first objectscomprise one or more microscope slides each containing a test sample ora tissue sample, wherein the first task further comprises one ofperforming processes in a histology laboratory or performing processesin a pathologist's office, and wherein the first task further comprisesa plurality of steps associated with each particular type of task;generating, with the computing system, a first image overlay, the firstimage overlay providing at least one of a graphical icon-basedinstruction, a text-based instruction, an image-based instruction, or ahighlighting-based instruction each indicative of one or moreinstructions to be presented to the first user to implement each of theplurality of steps of the sorting, categorizing, arranging, ororganizing of the one or more microscope slides containing the test ortissue samples; displaying, with the computing system and to the eyes ofthe first user through the AR headset, the generated first image overlaythat overlaps with the field of view of the eyes of the first user;tracking, with the computing system and using one or more second imagecapture devices that are disposed on the AR headset and that have afield of view that includes the eyes of the first user, the eyes of thefirst user to determine a direction in which the first user is looking;determining, with the computing system, the central portion of the fieldof view of the first user based at least in part on the determineddirection that the first user is looking in conjunction with the one ormore first images as captured by the one or more first image capturedevices; and selecting the at least one first object based on ananalysis of the central portion of the field of view of the first userand a verbal instruction by the first user to focus on the at least onefirst object; wherein the plurality of steps comprises: identifyingparticular one or more objects for the first user to interact with;identifying specific task-related step-by-step interactions, actuations,movements, transfers, or placements of the identified particular one ormore objects for one of performing processes in the histology laboratoryor performing processes in the pathologist's office; and identifyingspecific timings or sequences for each identified specific task-relatedstep-by-step interaction, actuation, movement, transfer, or placement ofthe identified particular one or more objects for accomplishing the oneof performing processes in the histology laboratory or performingprocesses in the pathologist's office; wherein, where the particular oneor more objects include instrumentation or equipment, the plurality ofsteps further comprises: identifying a model of the instrumentation orequipment for performing at least a portion of the one of performingprocesses in the histology laboratory or performing processes in thepathologist's office; and accessing documentation of the identifiedmodel of the instrumentation or equipment in response to user selectionof operation of the instrumentation or equipment; wherein identifyingspecific task-related step-by-step interactions, actuations, movements,transfers, or placements of the identified particular one or moreobjects comprises identifying specific task-related step-by-stepinteractions, actuations, movements, transfers, or placements of theidentified model of the instrumentation or equipment based oninformation in the accessed documentation and based on the userselection of operation of the instrumentation or equipment forperforming the at least the portion of the one of performing processesin the histology laboratory or performing processes in the pathologist'soffice; and wherein the generated first image overlay comprises one ormore image overlays of data corresponding to at least the identifiedparticular one or more objects, the identified specific task-relatedstep-by-step interactions, actuations, movements, transfers, orplacements of the identified particular one or more objects, and theidentified specific timings or sequences for each identified specifictask-related step-by-step interaction, actuation, movement, transfer, orplacement of the identified particular one or more objects foraccomplishing the one of performing processes in the histologylaboratory or performing processes in the pathologist's office.
 2. Amethod, comprising: receiving, with a computing system and from one ormore first image capture devices having a field of view of at least afirst portion of a work environment, one or more first images of the atleast the first portion of the work environment, the one or more firstimages overlapping with a field of view of eyes of a first user wearingan augmented reality (“AR”) headset; analyzing, with the computingsystem, the received one or more first images to identify one or morefirst objects disposed in the work environment, wherein the one or morefirst objects each comprises a radio frequency identifier (“RFID”) tagaffixed thereto, and wherein to identify the one or more first objectscomprises: identifying the one or more first objects based on the RFIDtag affixed to each first object in conjunction with identifying the oneor more first objects based on analysis of the received one or morefirst images; and filtering, based on a gesture or a selection by thefirst user, RFID tag information associated with objects outside of acentral portion of the field of view of the first user or outside of anarea defined by the gesture or the selection by the first user; queryingat least one database, with the computing system, to determine a firsttask associated with at least one first object among the identified oneor more first objects, wherein the first task comprises one ofperforming processes in a histology laboratory or performing processesin a pathologist's office, and wherein the first task further comprisesa plurality of steps associated with each particular type of task;generating, with the computing system, a first image overlay, the firstimage overlay providing one or more instructions to be presented to thefirst user to implement each of the plurality of steps of the first taskassociated with the at least one first object; and displaying, with thecomputing system and to the eyes of the first user through the ARheadset, the generated first image overlay that overlaps with the fieldof view of the eyes of the first user; wherein the plurality of stepscomprises: identifying particular one or more objects for the first userto interact with; identifying specific task-related step-by-stepinteractions, actuations, movements, transfers, or placements of theidentified particular one or more objects for one of performingprocesses in the histology laboratory or performing processes in thepathologist's office; and identifying specific timings and sequences foreach identified specific task-related step-by-step interaction,actuation, movement, transfer, or placement of the identified particularone or more objects for accomplishing the one of performing processes inthe histology laboratory or performing processes in the pathologist'soffice; wherein, where the particular one or more objects includeinstrumentation or equipment, the plurality of steps further comprises:identifying a model of the instrumentation or equipment for performingat least a portion of the one of performing processes in the histologylaboratory, or performing processes in the pathologist's office; andaccessing documentation of the identified model of the instrumentationor equipment in response to user selection of operation of theinstrumentation or equipment; wherein identifying specific task-relatedstep-by-step interactions, actuations, movements, transfers, orplacements of the identified particular one or more objects comprisesidentifying specific task-related step-by-step interactions, actuations,movements, transfers, or placements of the identified model of theinstrumentation or equipment based on information in the accesseddocumentation and based on the user selection of operation of theinstrumentation or equipment for performing the at least the portion ofthe one of performing processes in the histology laboratory orperforming processes in the pathologist's office; wherein the generatedfirst image overlay comprises one or more image overlays of datacorresponding to at least the identified particular one or more objects,the identified specific task-related step-by-step interactions,actuations, movements, transfers, or placements of the identifiedparticular one or more objects, and the identified specific timings andsequences for each identified specific task-related step-by-stepinteraction, actuation, movement, transfer, or placement of theidentified particular one or more objects for accomplishing the one ofperforming processes in the histology laboratory or performing processesin the pathologist's office.
 3. The method of claim 2, wherein thecomputing system comprises one of a processor disposed in the ARheadset, another computing system being disposed in the workenvironment, a remote computing system disposed external to the workenvironment and accessible over a network, or a cloud computing system.4. The method of claim 2, wherein the AR headset comprises one of a setof AR goggles, a pair of AR-enabled eyewear, an AR-enabled smartphonemounted in a headset, or an AR helmet.
 5. The method of claim 2, whereinthe AR headset comprises one or more of at least one earpiece or atleast one speaker, wherein the method further comprises: generating,with the computing system, one or more audio-based instructionscorresponding to each of at least one of the one or more instructions tobe presented to the user via the generated first image overlay; andpresenting, with the computing system and via one of the one or more ofthe at least one earpiece or the at least one speaker, the generated oneor more audio-based instructions in conjunction with displayingcorresponding each of at least one of the one or more instructions beingpresented to the user via the generated first image overlay as displayedto the eyes of the first user through the AR headset.
 6. The method ofclaim 2, wherein displaying the generated first image overlay to theeyes of the first user through the AR headset comprises one ofprojecting the generated first image overlay directly in the eyes of thefirst user, projecting the generated first image overlay on a projectionplane or surface in front of the eyes of the first user, projecting thegenerated first image overlay as a hologram in front of the eyes of thefirst user, displaying the generated first image overlay on atransparent or semi-transparent display screen of the AR headset that isdisposed in front of the eyes of the first user, or displaying thegenerated first image overlay superimposed over a continuous videorecording and display on a display screen of an AR-enabled smartphonemounted in the AR headset.
 7. The method of claim 2, wherein at leastone of the one or more first image capture devices is disposed on aportion of a housing of the AR headset.
 8. The method of claim 2,further comprising: receiving, with the computing system and from one ormore second image capture devices that are disposed within the workenvironment but external to the AR headset, one or more second images ofat least a second portion of the work environment; wherein analyzing,with the computing system, the received one or more first images toidentify one or more first objects disposed in the work environmentcomprises analyzing, with the computing system, the received one or morefirst images and the received one or more second images to identify theone or more first objects disposed in the work environment.
 9. Themethod of claim 2, wherein at least one of the one or more instructions,when displayed within the generated first image overlay, is superimposedover, displayed around, or displayed beside the at least one firstobject as viewed by the first user through the AR headset.
 10. Themethod of claim 9, wherein the at least one of the one or moreinstructions comprises at least one of a graphical icon-basedinstruction, a text-based instruction, an image-based instruction, or ahighlighting-based instruction.
 11. The method of claim 10, wherein: thegraphical icon-based instruction comprises at least one of a graphicalicon representing identification information associated with the atleast one first object, a graphical icon representing identificationinformation associated with a second object with which the at least onefirst object is intended to interact as part of the first task, acolored graphical icon distinguishing one first object from anotherfirst object among the at least one first object, a graphical iconcomprising a directional arrow representing a direction that the atleast one first object should be taken as part of the first task, or acolored graphical icon comprising a colored directional arrowdistinguishing a first direction that the at least one first objectshould be taken as part of the first task from a second direction that athird object should be taken as part of the first task; the text-basedinstruction comprises at least one of a floating text window comprisingtextual instructions corresponding to the first task or a surface textwindow that is superimposed on one of a table-top surface, a wallsurface, or an object surface and that comprises textual instructionscorresponding to the first task, wherein the floating text window, whendisplayed within the generated first image overlay, is displayed as afloating image beside the at least one first object or displayed as afloating image within the field of view of the eyes of the first user;the image-based instruction comprises at least one of an imagerepresenting identification information associated with the at least onefirst object, an image representing identification informationassociated with the second object with which the at least one firstobject is intended to interact as part of the first task, an imagecomprising the directional arrow representing the direction that the atleast one first object should be taken as part of the first task, animage comprising images of numbers or codes representing an order ofprocesses of the first task associated with the at least one firstobject, a magnified image of the at least one first object, athree-dimensional (“3D”) image or hologram, or an image of the at leastone first object superimposed over a targeted portion of the workenvironment indicating at least one of position, orientation, orconfiguration of the at least one first object intended for placement atthe targeted portion; and the highlighting-based instruction comprisesat least one of a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over the at least onefirst object, a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over a fourth object thatis related to the first task associated with the at least one firstobject, or a highlighting field that, when displayed within thegenerated first image overlay, is superimposed over the targeted portionof the work environment indicating intended placement of the at leastone first object at the targeted portion.
 12. The method of claim 2,wherein the one or more first objects each comprises a uniqueidentifier, wherein the unique identifier comprises a unique numericalidentifier, a unique alphanumeric identifier, a unique symbolicidentifier, a unique one-dimensional (“1D”) visual code, a uniquetwo-dimensional (“2D”) visual code, a unique three-dimensional (“3D”)visual code, a digital image of at least a portion of the first objectitself or of a representative example of the first object, or a uniqueidentifying feature or attribute, wherein analyzing, with the computingsystem, the received one or more first images to identify one or morefirst objects disposed in the work environment comprises identifying,with the computing system, the unique identifier that is associated witheach of the one or more first objects and that is displayed on at leasta portion of each of the one or more first objects.
 13. The method ofclaim 2, further comprising: illuminating, with a first light source,one or more of the at least one first object or the at least the firstportion of the work environment during at least one of the followingperiods: prior to the one or more first images being captured by the oneor more first image capture devices; or conjunction with the display ofthe generated first image overlay.
 14. The method of claim 13, whereinthe first light source is disposed on a portion of the AR headset. 15.The method of claim 2, further comprising: tracking, with the computingsystem and using one or more second image capture devices that aredisposed on the AR headset and that have a field of view that includesthe eyes of the first user, the eyes of the first user to determine adirection in which the first user is looking; and determining, with thecomputing system, the central portion of the field of view of the firstuser based at least in part on the determined direction that the firstuser is looking in conjunction with the one or more first images ascaptured by the one or more first image capture devices.
 16. The methodof claim 15, further comprising: in response to determining the centralportion of the field of view of the first user, identifying, with thecomputing system, at least one further object.
 17. The method of claim2, wherein the first task further comprises sorting, categorizing,arranging, or organizing the one or more first objects, wherein the oneor more objects comprise one or more microscope slides each containing atest sample or a tissue sample.
 18. The method of claim 2, wherein theone or more first objects comprises one of laboratory instrumentation,laboratory tools, sample transfer devices, puzzle components or pieces,sample reagents, sample containers, burners, coolers, mixers, samplepreparation tools, or sample transfer tools.
 19. The method of claim 2,wherein a plurality of users is tasked with performing the first task,wherein each user of the plurality of users wears an AR headset thatdisplays generated image overlays to each user that distinguishesobjects that one user is intended to interact with from objects thatother users among the plurality of users are intended to interact withas part of the first task.
 20. The method of claim 2, wherein the workenvironment comprises one of a laboratory, a construction site, amachine shop, a workshop, a factory, or a room.
 21. The method of claim2, further comprising: inferring a location of a plurality of objectsbased on placement of the identified particular one or more objectsadjacent to the plurality of objects.
 22. A system, comprising: acomputing system, comprising: at least one processor; and anon-transitory computer readable medium communicatively coupled to theat least one processor, the non-transitory computer readable mediumhaving stored thereon computer software comprising a set of instructionsthat, when executed by the at least one processor, causes the computingsystem to: receive, from one or more first image capture devices havinga field of view of at least a first portion of a work environment, oneor more first images of the at least the first portion of the workenvironment, the one or more first images overlapping with a field ofview of eyes of a first user wearing an augmented reality (“AR”)headset; analyze the received one or more first images to identify oneor more first objects disposed in the work environment, wherein the oneor more first objects each comprises a radio frequency identifier(“RFID”) tag affixed thereto, and wherein to identify the one or morefirst objects comprises: identifying the one or more first objects basedon the RFID tag affixed to each first object in conjunction withidentifying the one or more first objects based on analysis of thereceived one or more first images; and filtering, based on a gesture ora selection by the first user, RFID tag information associated withobjects outside of a central portion of the field of view of the firstuser or outside of an area defined by the gesture or the selection bythe first user; query at least one database to determine a first taskassociated with at least one first object among the identified one ormore first objects, wherein the first task comprises one of performingprocesses in a histology laboratory or performing processes in apathologist's office, and wherein the first task further comprises aplurality of steps associated with each particular type of task;generate a first image overlay, the first image overlay providing one ormore instructions to be presented to the first user to implement each ofthe plurality of steps of the first task associated with the at leastone first object; and display, to the eyes of the first user through theAR headset, the generated first image overlay that overlaps with thefield of view of the eyes of the first user; wherein the plurality ofsteps comprises: identifying particular one or more objects for thefirst user to interact with; identifying specific task-relatedstep-by-step interactions, actuations, movements, transfers, orplacements of the identified particular one or more objects for one ofperforming processes in the histology laboratory or performing processesin the pathologist's office; and identifying specific timings orsequences for each identified specific task-related step-by-stepinteraction, actuation, movement, transfer, or placement of theidentified particular one or more objects for accomplishing the one ofperforming processes in the histology laboratory or performing processesin the pathologist's office; wherein, where the particular one or moreobjects include instrumentation or equipment, the plurality of stepsfurther comprises: identifying a model of the instrumentation orequipment for performing at least a portion of the one of performingprocesses in the histology laboratory or performing processes in thepathologist's office; and accessing documentation of the identifiedmodel of the instrumentation or equipment in response to user selectionof operation of the instrumentation or equipment; wherein identifyingspecific task-related step-by-step interactions, actuations, movements,transfers, or placements of the identified particular one or moreobjects comprises identifying specific task-related step-by-stepinteractions, actuations, movements, transfers, or placements of theidentified model of the instrumentation or equipment based oninformation in the accessed documentation and based on the userselection of operation of the instrumentation or equipment forperforming the at least the portion of the one of performing processesin the histology laboratory or performing processes in the pathologist'soffice; and wherein the generated first image overlay comprises one ormore image overlays of data corresponding to at least the identifiedparticular one or more objects, the identified specific task-relatedstep-by-step interactions, actuations, movements, transfers, orplacements of the identified particular one or more objects, theidentified specific timings or sequences for each identified specifictask-related step-by-step interaction, actuation, movement, transfer, orplacement of the identified particular one or more objects foraccomplishing the one of performing processes in the histologylaboratory or performing processes in the pathologist's office, and anexpected wait time of availability of the instrumentation or equipmentfor the first user based on monitoring of use of the instrumentation orequipment by other users.
 23. An apparatus, comprising: at least oneprocessor; and a non-transitory computer readable medium communicativelycoupled to the at least one processor, the non-transitory computerreadable medium having stored thereon computer software comprising a setof instructions that, when executed by the at least one processor,causes the apparatus to: receive, from one or more first image capturedevices having a field of view of at least a first portion of a workenvironment, one or more first images of the at least the first portionof the work environment, the one or more first images overlapping with afield of view of eyes of a first user wearing an augmented reality(“AR”) headset; analyze the received one or more first images toidentify one or more first objects disposed in the work environment;query at least one database to determine a first task associated with atleast one first object among the identified one or more first objects,wherein the first task comprises one of performing processes in ahistology laboratory or performing processes in a pathologist's office,and wherein the first task further comprises a plurality of stepsassociated with each particular type of task; generate a first imageoverlay, the first image overlay providing one or more instructions tobe presented to the first user to implement each of the plurality ofsteps of the first task associated with the at least one first object;and display, to the eyes of the first user through the AR headset, thegenerated first image overlay that overlaps with the field of view ofthe eyes of the first user; wherein the plurality of steps comprises:identifying particular one or more objects for the first user tointeract with; identifying specific task-related step-by-stepinteractions, actuations, movements, or transfers, and placements, basedon meeting of a specified percentage condition of usage that includespartial usage, of the identified particular one or more objects for oneof performing processes in the histology laboratory or performingprocesses in the pathologist's office; and identifying specific timingsor sequences for each identified specific task-related step-by-stepinteraction, actuation, movement, transfer, or placement of theidentified particular one or more objects for accomplishing the one ofperforming processes in the histology laboratory or performing processesin the pathologist's office; wherein, where the particular one or moreobjects include instrumentation or equipment, the plurality of stepsfurther comprises: identifying a model of the instrumentation orequipment for performing at least a portion of the one of performingprocesses in the histology laboratory, or performing processes in thepathologist's office; and accessing documentation of the identifiedmodel of the instrumentation or equipment in response to user selectionof operation of the instrumentation or equipment; wherein identifyingspecific task-related step-by-step interactions, actuations, movements,transfers, or placements of the identified particular one or moreobjects comprises identifying specific task-related step-by-stepinteractions, actuations, movements, transfers, or placements of theidentified model of the instrumentation or equipment based oninformation in the accessed documentation and based on the userselection of operation of the instrumentation or equipment forperforming the at least the portion of the one of performing processesin the histology laboratory or performing processes in the pathologist'soffice; and wherein the generated first image overlay comprises one ormore image overlays of data corresponding to at least the identifiedparticular one or more objects, the identified specific task-relatedstep-by-step interactions, actuations, movements, transfers, orplacements of the identified particular one or more objects, and theidentified specific timings or sequences for each identified specifictask-related step-by-step interaction, actuation, movement, transfer, orplacement of the identified particular one or more objects foraccomplishing the one of performing processes in the histologylaboratory or performing processes in the pathologist's office.